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EXECUTIVE  SUMMARY 


Air  Force  Manual  (AFM)  50-2  defines  Instructional  System  Design 
(ISO)  as  a  "systematic  procedure  for  assuring  application  of  instruc¬ 
tional  technology  to  course  planning  and  development."  Task  analysis,  a 
critical  front-end  activity  in  the  ISD  methodology,  is  the  process  of 
partitioning  job  tasks  into  their  component  subtasks  and  identifying  the 
skills  and  knowledges  required  to  support  task  performance.  The  output 
of  the  task  analysis  activity  may  be  thought  of  as  a  specification  which 
defines  the  content  of  an  instructional  program.  Emphasis  is  placed 
upon  identifying  only  those  skills  and  knowledges  which  must  be  taught 
to  support  task  performance  because  overtraining  and  undertraining  are 
extremely  costly  and  wasteful. 

This  research  was  initiated  in  response  to  a  Request  for  Personnel 
Research  generated  by  the  Air  Training  Command  (ATC)  which  noted  that  "a 
wide  range  of  nonstandardized  task  analysis  procedures  are  in  use 
throughout  the  training  community."  It  was  felt  that  implementation  of 
a  standardized  procedure  for  identifying  essential  subtasks  and  support¬ 
ing  skills  and  knowledges  held  considerable  potential  for  increasing 
training  efficiency  and  reducing  training  costs. 

In  Phase  I,  ATC  training  development  and  management  personnel 
and  their  counterparts  within  the  Military  Airlift  Command  (MAC), 
the  Strategic  Air  Command  (SAC),  and  the  Tactical  Air  Command  (TAC) 
were  interviewed  regarding  task  analysis  procedures  currently  in  use. 
Twenty-five  groups,  ranging  in  size  from  two  to  nine  individuals,  at 
eight  Air  Force  (AF)  installations  participated  in  the  survey.  Current 
ATC  task  analysis  procedures  and  ISD  training  and  guidance  documentation 
were  reviewed  and  evaluated.  Recommendations  for  improving  the  ATC  task 
analysis  effort  included  developing  and  field  testing  a  simplified  task 
analysis  procedure  and  documentation  system. 

In  Phase  II,  a  standardized  task  analysis  procedure  was  specified 
and  a  prototype  task  analysis  handbook  was  prepared.  A  two-stage  field 
test  was  conducted.  Stage  1  consisted  of  preliminary  tryouts  conducted 
to  obtain  information  useful  for  revising  the  handbook  prior  to  formal 
evaluation.  Thirty-five  training  development  personnel  at  three  AF 
bases  either  utilized  the  prototype  procedures  to  conduct  a  task  analy¬ 
sis  or  critically  reviewed  the  handbook.  A  sizable  number  of  sugges¬ 
tions  for  improving  the  handbook  were  generated  and  numerous  revisions 
were  made.  Stage  2,  feasibility  testing,  was  devoted  to  assessing  the 
simplicity,  reliability,  validity,  and  time-efficiency  of  handbook  task 
analysis  procedures.  Sixty-five  training  development  personnel  at  three 
ATC  Technical  Training  Centers  (TTCs )  .partici pated  in  feasibility 
testing.  Some  utilized  handbook  procedures  to  conduct  task  analyses, 
others  reviewed  and  critiqued  the  handbook.  A  substantial  majority  of 
the  participants  felt  that  the  handbook  procedures  were  simple  to  use 
and  would  require  less  time  to  complete  than  the  current  procedures. 
More  importantly,  evaluation  of  completed  task  analyses  indicated  that 
handbook  procedures  were  reliable  and  valid. 
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In  Phase  III,  a  final  draft  cf  the  task  analysis  handbook  was 
prepared,  reviewed  in  conference  v.:th  intended  users  and  ATC  manayenient 
personnel,  and  revised.  The  handbook  has  been  published  as  AFHRL- 
TR-79-45  (II).  A  preliminary  design  for  an  automated  storage  and 
retrieval  system  for  ATC  task  analysis  documentation  was  then  prepared. 
Additionally,  two  technology  transfer  seminars  were  conducted  to  assist 
AF  personnel  in  applying  handbook  task  analysis  procedures. 
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SECTION  I 


INTRODUCTION 


BACKGROUND 

Si  nee  the  AF  developed  its  first  major  instructional  system  in 
1965,  the  systems  approach  to  training  has  received  considerable  empha¬ 
sis  within  the  Department  of  Defense  and  in  the  civilian  sector.  The 
issuance  of  AF  Manual  (AFM)  50-2,  Instructional  System  Design.  AF 
Pamphlet  (AFP)  50-58,  Handbook  for  Designers  of  Instructional  Systems, 
and  AF  Regulation  50-8, "Instructional  Systems  Development  (iSP),  witnes- 
sed  a  realization  on  the  part  of  the  AF  that  application  of  modern 
instructional  technologies  might  yield  substantial  improvements  in  the 
effectiveness  and  efficiency  of  AF  training  programs.  These  documents 
place  considerable  emphasis  on  achieving  close  correspondence  between 
training  program  content  and  job  performance  requirements. 

The  Occupational  Surveys  (OSs  )  produced  by  the  USAF  Occupational 
Measurement  Center  (OMC)  are  an  important  information  source  for  accom¬ 
plishing  job  analysis  and  specifying  job  perfonnance  requirements  within 
the  context  of  AF  technical  training.  However,  more  detailed  informa¬ 
tion  about  job  performance  subtasks  and  supporting  skills  and  knowledges 
is  required  to  design  effective  and  efficient  training  programs.  Task 
analysis,  the  process  by  which  that  detailed  information  is  generated, 
is  the  most  important  step  in  designing  an  instructional  system.  The 
process  by  which  a  skilled  instructional  designer  identifies  the  major 
procedural  steps  within  a  task  and  then  makes  inferences  about  required 
skills  and  knowledges  is  not  well  defined.  Additionally,  those  in  the 
ATC  who  are  responsible  for  conducting  and  documenting  task  analyses  are 
Subject-Matter  Specialists  (SMSs),  not  educational  technologists.  The 
implementation  of  a  simplified  task  analysis  procedure/documentation 
system  and  of  a  computer-based  task  analysis  data  bank  may  offer  signi¬ 
ficant  economies  in  the  design  and  revision  of  technical  training 
courses.  A  standardized  task  analysis  procedure  would  help  insure  that 
course  content  decisions  are  made  on  the  basis  of  job  performance 
requirements  as  tempered  by  training  situation  constraints;  and  a 
computer-based  data  bank  would  provide  a  means  of  storing,  retrieving, 
updating,  and  disseminating  current  task  analysis  information.  Ulti¬ 
mately,  these  economies  might  be  expected  to  manifest  themselves  in  the 
form  of  more  effective  and  less  costly  training. 

STUDY  OBJECTIVES  AND  APPROACH 


The  primary  objective  of  this  study  was  to  develop  and  field  test 
a  simpl e-to-use,  reliable,  valid,  and  cost-effective/time-efficient 
task  analysis  procedure  for  application  by  the  ATC  training  development 
personnel  responsible  for  the  design  and  conduct  of  technical  training 
courses.  A  secondary  objective  was  to  make  recommendations  regarding 
the  feasibility  and  utility  of  implementing  a  computer-based  task 
analysis  data  bank  and  to  submit  a  preliminary  data  bank  design  for 
consideration.  End  items  included; 
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1.  A  handbook  detailing  a  standardized  task  analysis  procedure 
that  will  provide  an  acceptable  degree  of  uniformity  and 
quality  control  across  ATC  task  analysis  efforts. 

2.  A  systems  analysis  of  present  and  future  AF  task  analysis 
requirements,  including  a  preliminary  design  for  a  task  analy¬ 
sis  data  bank. 

The  investigative  approach  employee  in  this  study  was  straight¬ 
forward.  In  Phase  I,  current  ATC  task  analysis  procedures  were  charac¬ 
terized  and  evaluated.  Recommendations  for  improving  the  task  analysis 
effort  were  then  proposed.  In  Phase  II,  a  standardized  procedure 
was  specified  and  a  prototype  handbook  was  developed.  It  was  field 
tested  at  ATC  TTCs  and  revised  on  the  basis  of  field  test  results.  In 
Phase  III,  a  final  draft  of  the  task  analysis  handbook  was  prepared, 
reviewed  in  conference  with  intended  users  and  management  personnel,  and 
revised  prior  to  finalization.  A  preliminary  plan  for  an  automated 
storage  and  retrieval  system  for  task  analysis  data  was  then  developed. 
Additionally,  two  technology  transfer  seminars  were  conducted  to  assist 
AF  personnel  in  applying  handbook  task  analysis  procedures.  It  should 
be  noted  that  ATC  training  development  and  management  personnel  were 
continuously  involved  in  the  design,  testing,  and  revision  of  the 
handbook  to  insure  that  it  was  useful,  and  to  maximize  the  probability 
that  it  would  be  accepted  and  implemented. 

GLOSSARY  OF  TERMS 

A  glossary  of  technical  terms  and  their  acronyms  used  in  this 
report  is  presented  in  Appendix  H. 


SECTION  II 


METHOD 


SURVEY  OF  CURRENT  TASK  ANALYSIS  PROCEDURES 


Interviews 


Research  interviews  were  conducted  to  gain  insight  into  the  task 
analysis  procedures  currently  being  utilized  in  the  AF  technical  train¬ 
ing  community.  The  interview  consisted  of  a  set  of  specific  questions, 
each  with  an  open  response  set.  Also  known  as  the  open-ended  or  free- 
response  format,  this  type  of  interview  was  chosen  because  it  insures 
that  relevant  areas  of  inquiry  are  addressed  in  sufficient  depth,  by 
permitting  interviewees  to  respond  freely  and  to  elaborate  in  their  own 
words.  It  also  allows  the  interviewer  to  selectively  probe  more  deeply 
into  interesting  and  novel  responses.  Further,  the  informal  atmosphere 
created  by  this  type  of  interview  encourages  interviewees  to  be  both 
cooperative  and  candid. 

Development  of  the  Interview 

Prior  to  developing  the  interview,  a  review  of  current  military  and 
civilian  task  analysis  procedures  and  manuals  was  accomplished.  In 
addition,  conferences  and  discussions  were  held  vyith  cognizant  personnel 
from  the  sponsoring  agency  and  ATC.  Based  upon  these  meetings  and  the 
document  review,  the  following  specific  areas  of  inquiry  were  identified 
for  inclusion  in  the  interview: 

1.  The  relative  percentage  of  time  spent  revising  existing  courses 
versus  developing  new  courses  and  manpower/resource  expenditure 
accounting. 

2.  The  process  utilized  for  determining  tasks  which  require 
instruction. 

3.  The  process  utilized  for  determining  proficiency  requirements. 

4.  The  manner  in  which  OS  and  Occupational  Survey  Report  (OSR) 
data  are  utilized. 

5.  The  procedures  used  for  subtask  identification. 

6.  The  procedures  used  for  skill/knowledge  analyses. 

7.  The  manner  in  which  subtask  and  skill/knowledge  analyses  are 
validated  (verified). 

8.  The  adequacy  of  reviews,  student  critiques,  and  field  evalu¬ 
ations  for  determining  training  inadequacy/excessiveness. 
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9.  The  familiarity  with  and  judged  adequacy  of  AFM  50-2,  AFP  50-58 
and  the  Interservice  Procedures  for  Instructional  System 
Develof)ment  (IPISD). 

10.  The  extent  of  formal  training  in  ISO  and  its  judged  adequacy. 

11.  The  overall  opinions  regarding  the  AF  ISO  program. 

12.  The  problems  associated  with  implementing  the  AF  ISO  model. 

In  addition  to  the  interview,  a  decision  was  made  to  secure  copies  of 
task  analysis  and  ISO  worksheets  and  locally  produced  ISO  guidance 
documents,  when  interviewees  indicated  that  a  formal  documentation 
system  or  local  procedures  were  utilized. 

Pretest  of  the  Interview 

The  interview  in  its  original  form,  was  pretested  on  training 
managers,  training  specialists  and  SMSs  at  Sheppard  TTC.  This  pretest¬ 
ing  indicated  necessary  refinements.  Certain  questions  were  redundant 
and,  therefore,  eliminated.  Other  questions  required  rephrasing  for  the 
purpose  of  enhancing  clarity.  Finally,  the  pretest  indicated  that  a 
reordering  of  some  of  the  questions  was  necessary  in  order  to  obtain 
continuity  in  the  flow  of  thoughts  and  ideas  that  were  being  elicited 
from  the  interviewees. 

Conduct  of  the  Interviews 

Three  interviewers,  all  members  of  the  Engineering  Psychology 
Department  of  McDonnell  Douglas  Astronautics  Company  -  St.  Louis  (MDAC- 
St.  Louis),  collected  the  data  in  the  field.  Each  of  these  interviewers 
had  extensive  experience  in  both  structured  and  unstructured  interview¬ 
ing.  In  addition,  each  interviewer  was  generally  familiar  with  the  AF 
ISO  program  and  had  reviewed  pertinent  documentation  in  the  area  of 
task  analysis. 

Those  interviewed  included  a  full  range  of  training  development 
personnel,  including  military  and  civilian  education  specialists,  ISD 
technicians,  and  master  instructors  who  had  been,  or  were  currently, 
involved  with  task  analysis  efforts  at  the  five  ATC  TTCs.  In  addition, 
training  development  personnel  from  the  3306th  Test  and  Evaluation 
Squadron  at  Edwards  Air  Force  (AFB);  the  USAF  School  of  Aerospace 
Medicine,  Brooks  AFB;  and  the  School  of  Health  Care  Sciences,  Sheppard 
AFB,  were  also  interviewed.  Finally,  training  development  personnel 
from  the  MAC,  the  TAC,  and  the  SAC  were  interviewed  regarding  their 
current  ISD  and  task  analysis  efforts. 

Review  of  ISD  Guidance  Documentation 

In  addition  to  soliciting  the  opinions  of  training  development 
personnel  regarding  the  ISD  and  task  analysis  guidance  provided  in  AFM 
50-2  and  AFP  50-58,  the  study  team  conducted  an  independent  review  and 


assessment  of  these  documents.  This  assessment,  necessarily  subjective, 
was  conducted  along  two  primary  dimensions: 


1.  The  inherent  logic  of  AF  task  analysis  procedures  and  the  ISO 
model  in  which  they  are  embedded. 

2.  The  usability  and  readability  of  the  manual  and  pamphlet  that 
document  those  procedures  and  that  model . 

The  issue  dates  for  the  documents  reviewed  were  31  July  1975  for  AFM 
50-2,  and  15  July  1973  for  AFP  50-58.  It  should  be  noted  that  both 
documents  have  subsequently  been  revised  by  ATC  based  on  field  survey 
results. 


Review  of  AF  ISO  Courses 


The  assessment  of  AF  ISO  training  was  based  upon  a  review  of  course 
control  documents  and  student  study  guides  and  workbooks  furnisiied  by 
the  Instructor  Training  Branch  of  the  3700th  Technical  Training  Wing, 
Sheppard  AFB.  Specific  courses  examined  included:  3AIR75100,  Technical 
Instructor,  and  3AIR75130-X,  Instructional  System  Development.  The 
primary  intent  of  the  review  was  to  evaluate  AF  ISD  training  generally 
but  with  particular  emphasis  on  those  instructional  units  dealing 
specifically  with  task  analysis.  A  secondary  objective  was  to  generate 
an  information  base  for  judging  the  validity  of  course  graduate  comments 
gathered  during  the  on-site  interviews. 

Course  (unit)  review  criteria  clustered  along  three  major  dimen¬ 
sions: 

1.  The  clarity  and  appropriateness  of  instructional  objectives. 

2.  The  adequacy  of  the  information  and  practice  exercises  pre¬ 
sented. 

3.  The  clarity  of  the  interrelationships  among  major  concepts. 

A  truly  rigorous  evaluation  of  AF  ISD  training  courses  would,  of  course, 
have  required  attendance  at  and  active  participation  in  each  of  the 
subject  courses.  Unfortunately,  budget  constraints  did  not  permit 
this  type  of  evaluation. 

RESULTS 

Current  Procedures  Survey:  Technical  Training  Centers 

Relative  Percentage  of  Time  Spent  Revising  Existing  Courses  Versus 
Developing  New  Courses 

0  Of  the  time  spent  in  task  analysis  activities,  the  overwhelming 
proportion  was  expended  in  accomplishing  analyses  to  support 
revision  of  existing  courses. 
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0  Training  development  personnel  deemed  existing  task  analysis 
procedures  more  useful  for  developing  new  courses  than  for 
revising  existing  courses. 

Nanpower/Resource  Expenditure  Accounting 

0  Training  managers  and  training  development  personnel  did  not 
record  time  spent  or  number  of  personnel  involved  in  task 
analysis  (or  ISO)  efforts. 

Determination  of  Which  Tasks  Require  Training 

0  Training  development  personnel  assumed  that  all  tasks  listed  in 
the  Speciality  Training  Standard  (STS)  or  Course  Training 
Standard  (CTS)  required  training. 

0  The  OS  was  a  supplementary  source  of  information  used  to  "verify" 
STS/CTS  lists. 

Determination  of  Proficiency  Requirements 

0  Training  development  personnel  examined  and  interpreted  STS/CTS 
proficiency  codes  to  define  required  proficiency  levels. 

0  A  few  training  development  groups  utilized  the  task  difficulty 
data  in  the  OS  to  determine  proficiency  requirements. 

Use  of  OS  in  Task  Analysis  Process 

0  A  majority  of  training  development  personnel  indicated  that  OS 
infonnation  was  useful  in  performing  task  analysis. 

0  Most  of  those  interviewed  expressed  concern  about  the  currency 
and  validity  of  OSs. 

0  Most  groups  were  unaware  of  the  special  purpose  OS  analyses  that 
can  be  accomplished  by  the  OMC. 

Subtask  Identification 

0  Instructor  and  SMS  experience  was  considered  the  most  important 
source  of  information  for  breaking  a  major  task  into  its  com¬ 
ponent  subtasks. 

Ski  11 /Knowledge  Analysis 

0  The  skills  and  knowledges  required  to  support  subtask  and  task 
performance  were  inferred  by  instructors  and  SMSs. 

Adequacy  of  Guidance  Documentation  (AFM  50-2  and  AFP  50-53) 


0  AFM  50-2  and  AFP  50-58  were  not  widely  distributed. 
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0  Training  development  personnel  suggested  that  AFP  50-58  proce¬ 
dures  were  too  complex,  required  too  much  paperwork,  were  in 
conflict  with  ATC  regulations  and  policies,  were  oriented 
primarily  towards  development  of  new  courses,  and  were  too 
oriented  toward  aircraft  and  equipment. 

0  Most  of  those  interviewed  indicated  that  the  documents  would  be 
more  useful  if  time  and  resources  permitted  accompl  isfmient  of 
required  effort. 

Documentation  of  Task  Analyses 

0  Relatively  few  groups  had  any  formal,  systematic  way  of  docu¬ 
menting  their  task  analyses. 

Verification  Procedures 

0  Task  analyses  were  verified  as  part  of  the  course  implementation 
process  (if  the  course  worked  well,  then  task  analyses  had 
been  properly  done). 

Assessmetit  of  Training  Inadequac.y/Excessi veness 

0  Training  development  personnel  most  often  relied  on  field 
evaluation  data  for  determining  training  deficiences  and  exces¬ 
ses,  but  none  were  completely  satisfied  with  the  validity  of 
these  data. 

0  Current  procedures  for  internal  reviews  of  course  control 
documents,  curriculum  materials,  resources,  and  test  items  were 
generally  deemed  adequate. 

0  Student-generated  course  critiques  were  viewed  with  suspicion 
and  considered  an  inadequate  index  of  training  inadequacy  and 
excessiveness. 

Adequacy  of  Formal  ISP  Training 

0  The  two  ISD  courses  that  the  majority  of  task  analysts  complete 
(3AIR75100,  Technical  Instructor,  and  3AIR75130-X,  Instructional 
System  Development)  were  considered  inadequate  by  the  majority 
of  interviewees: 

-  3AIR75100  did  not  achieve  sufficient  depth. 

-  3AIR75130-X  was  too  detailed,  boring,  and  dry. 

Quality  of  and  Problems  Associated  with  AF  ISD  Program 

0  Time  and  resource  constraints  impede  successful  implementation 
of  the  ISD  model. 

0  The  ISD  model  is  too  difficult  to  understand,  too  complex  to 
implement,  and  requires  too  much  paperwork. 


Current  Procedures  Survey:  Major  Commands  (MAJCOHS 


Approach  to  ISP 

0  ISO  teams  were  generally  comprised  of  SMSs,  civilian  training 
specialists,  a  military  education/training  officer,  and  an 
enlisted  ISO  technician. 

0  SMSs  were  primarily  responsible  for  completing  task  analyses. 
Task  Analysis  Methods 

0  TAC  and  SAC  relied  heavily  on  Mager's  Criterion-Referenced 
Instruction  (CRI )  approach  to  task  analysis. 

0  MAC  utilized  a  four-step  process: 

-  List  tasks. 

-  Identify  training  requirements. 

-  Develop  criterion  objectives  and  subobjectives  or  subtasks. 

-  Identify  teaching  points  based  on  supporting  skills  and 
knowledges. 

Infoniiation  Sources 


0  Training  development  personnel  utilized  multiple  information 
sources  to  support  task  analysis  efforts: 

-  OSs. 

-  Technical  orders  (T.O.s)  and  manuals. 

-  Existing  training  materials. 

-  SMS  inputs. 

Guidance  Documentation 


0  Training  development  personnel  used  the  following  guidance 
documents: 

-  CRI  materials. 

-  AFM  50-2. 

-  AFP  50-58. 

-  IPISD. 

-  Locally  developed  materials. 
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Documentation  of  Task  Analyses 


0  All  MAJCOMs  produced  task  analysis  documentation  detailing 
tasks,  subtasks,  and  supporting  skills  and  knowledges. 

Validation  of  Task  Analyses 

0  MAJCOMs  relied  upon  SMSs  or  the  local  Standardization/  Evalu¬ 
ation  Branch  to  review  the  task  analyses  for  accuracy,  complete¬ 
ness,  and  appropriateness. 

0  Field  evaluation  reports  (FERs)  were  also  used  to  verify  task 
analyses. 

Study  Team  Review  of  AFM  50-2  and  AFP  50-58 


AFM  50-2 


A  review  of  AFM  50-2  indicated  that  it  contains  some  useful  task 
analysis  guidance  but  that  a  considerable  degree  of  sophistication 
would  be  required  to  isolate  and  interpret  it.  The  document  appeared  to 
be  more  useful  for  training  managers  than  for  training  development 
personnel  in  that  it  provides  a  thorough  overviev/  of  the  ISO  process. 
It  describes  what  must  be  done,  but  does  not  provide  detailed  guidance 
regarding  how  things  are  to  be  done.  In  defense  of  AFM  50-2,  it  was 
noted  that  it  v/as  not  intended  to  be  a  procedures  manual. 

AFP  50-58  (Volumes  I  and  II  ) 

The  ISO  model  and  task  analysis  procedures  presented  in  AFP  50-58, 
Volumes  I  and  II  (Chapters  1,  2,  3,  and  5),  are  logically  consistent. 
The  documents  provide  a  good  orientation  to  ISO  and  a  good  overview  of 
the  major  steps  that  comprise  the  model.  However,  the  orientation 
and  overview  of  the  task  analysis  process  was  considered  to  be  only 
marginally  adequate. 

Each  step  in  the  task  analysis  process  presented  would  result  in 
the  generation  of  information  necessary  to  perform  the  succeeding  step. 
In  addition,  the  procedures  set  forth  for  each  step  are  adequate  to 
operate  on  the  information  or  data  generated  in  the  preceeding  step. 
The  documents  also  provided  explanations  and  examples  that  were  adequate 
to  permit  an  understanding  of  the  task  analysis  process  described. 
Finally,  the  relationship  between  intennediate  and  accountable  end-item 
documents  was  generally  clear. 

Volumes  I  and  II  suffer  considerably  on  the  usability/readability 
dimension.  An  attempt  is  made  to  organize  the  concepts  and  proce¬ 
dures  to  facilitate  understanding  and  application.  However,  many 
of  the  concepts  are  inherently  difficult  individually  and  certainly 
collectively.  Comprehensiveness  is  often  achieved  at  the  expense  of 


clarity  of  presentation.  Detailed  descriptions  of  management  concerns 
are  often  embedded  within  the  task  analysis  guidance  and  tend  to  obs¬ 
cure,  and  sometimes  overpower  it.  Main  points  are  generally  difficult 
to  identify. 

The  writing  style  is,  for  the  most  part,  straightforward  and 
comprehensible.  However,  the  heavy  dependency  on  figures  and  tables 
results  in  extreme  fragmentation  of  the  text.  For  example,  of  the  88 
pages  that  comprise  Chapter  2  and  3,  70  pages  contain  figures  and/or 
tables.  Twenty-eight  of  these  pages  contain  full-page  figures  or 
tables,  and  42  contain  half-page  figures  or  tables.  This  fragmentation 
can  discourage  even  the  most  interested  reader.  For  that  reason, 
the  reading  demands  are  considered  excessive. 

Flow  diagrams  are  the  primary  mechanism  for  providing  an  integrated 
overview  of  and  showing  the  interrelationships  among  the  activities  that 
comprise  Step  1  and  2  of  the  ISD  model.  Additionally,  flow  diagrams 
are  used  to  depict  some  of  the  more  complex  step-by-step  procedures 
described.  This  type  of  figure  requires  a  special  type  of  literacy  that 
most  users  would  not  possess.  The  figures  and  tables,  while  generally 
adequate  in  the  sense  that  they  contain  useful  information,  do  contri¬ 
bute  substantially  to  the  textual  fragmentation  problem  mentioned 
previously. 

In  conclusion,  there  is  some  useful  task  analysis  guioance  in  AFP 
50-58,  but  a  considerable  degree  of  sophistication  and  experience  would 
be  required  to  isolate  and  interpret  it.  It  would  seem  unreasonable  to 
expect  that  a  SME  or  group  of  SMEs  could  implement  that  task  analysis 
process  described.  An  individual  with  considerable  backgound  in  ISD 
would  have  to  translate  the  task  analysis  process  described  into  a  plan 
of  action  to  cover  a  particular  course  design  or  redesign  effort;  and, 
further,  would  have  to  actively  participate  in  the  effort. 

Study  Team  Review  of  AF  ISD  Training 
Course  3AIR75100.  Technical  Instructor 

The  16-hour  ISD  block  of  the  3AIR75100  course  exhibited  both 
strengths  and  weaknesses.  The  instructional  intent  was  clearly  con¬ 
veyed,  and  the  supporting  objectives  were  appropriate.  The  level  of 
difficulty  was  also  deemed  appropriate  for  the  student  population. 
Additionally,  the  practice  exercises  and  format  of  the  course  materials 
we^'e  judged  to  be  adequate.  However,  these  strengths  were  attenuated 
somewhat  by  the  cef iciencies.  Specifically,  the  study  guide  and  cor- 
5"csponding  sections  of  the  workbook  did  not  accurately  portray  the 
true  nature  of  the  ISO  model.  Important  aspects  of  the  model  were 
omitted,  and  the  interaction  of  the  model's  five  steps  was  not  well 
addressed. 
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Course  3AIR75130-X.  Instructional  System  Development 


The  3AIR75130-X  course  was  deemed  satisfactory  with  regard  to 
clarity  and  appropriateness  of  instructional  objectives,  accuracy  of 
information  presented,  and  adequacy  of  instructional  logic.  However, 
the  difficulty  level  of  the  course,  along  with  the  use  of  adjunctive 
programming  techniques,  may  overwhelm  all  but  the  brightest  and  most 
highly  motivated  students. 

CONCLUSIONS  AND  RECOMMENDATIONS 


The  survey  findings  revealed  a  multitude  of  general  and  specific 
problems  with  regard  to  ATC's  task  analysis  and  ISO  efforts.  These 
problems  seemed  to  cluster  in  three  areas;  therefore,  the  conclusions 
and  recommendations  are  organized  and  reported  in  three  sections. 
The  first  section  addresses  the  area  of  administration  and  management  of 
the  ATC  task  analysis  and  ISD  efforts.  The  second  section  encompasses 
training  of  ATC  personnel  who  participate  in  such  efforts.  The  third 
section  presents  conclusions  and  recommendations  that  deal  with  the 
nature  of  documentary  guidance  currently  available  to  ATC  training 
development  personnel. 


Administration  and  Management 
ATC  Task  Analysis  Effort 

There  was  considerable  variation  in  the  rigor  with  v^hich  the  groups 
at  various  centers,  and  even  branches  within  groups,  accompl i slied  and 
documented  task  analyses.  In  all  cases,  the  amount  of  rigor  achieveo 
seemed  directly  proportional  to  management's  cornmitment  to  and  experi¬ 
ence  with  ISD.  This  wide  variation  in  commitment  suggested  the  need  for 
a  better  orientation  to  and  improved  training  in  the  ISD  process  and 
task  analysis  procedures  for  all  levels  of  management  and  supervision. 
This  perceived  need  will  be  addressed  at  greater  length  in  one  of  the 
recommendations  that  follows. 

The  task  analysis  procedures  and  documentation  methods  currently 
utilized  at  the  centers  were  widely  variant.  Documentation  produced  in 
response  to  inquiries  regarding  how  the  results  of  task  analyses  were 
recorded  ranged  from  Plans  of  Instruction  (POIs)  to  fairly  detailed  ISD 
worksheets,  most  of  which  were  locally  designed.  Under  the  circum¬ 
stances  that  prevailed  at  the  time  of  the  survey,  quality  control  of  tlie 
task  analysis  effort  across  branches  within  the  same  group  would  have 
been  difficult  and  an  integrated  quality  control  program  across  centers 
would  have  been  virtually  impossible.  Therefore,  an  attempt  to  develop 
and  implement  a  standardized  task  analysis  procedure  and  documentation 
system  for  application  at  all  centers  seemed  a  worthwhile  pursuit. 
Standardization  was  expected  to  permit  initiation  of  a  rigorous  quality 
control  program  directed  toward  maximizing  management's  support  of  the 
task  analysis  effort.  Increased  management  support  might  have  the 
beneficial  effect  of  improving  the  attitudes  and  motivation  of  training 
development  personnel  at  the  centers. 
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The  issue  of  accountability  was,  of  course,  closely  related  to 
quality  control.  Once  again,  the  survey  showed  that  no  individual  or 
group  of  individuals  was  ultimately  accountable  for  the  task  analysis 
within  the  present  organization.  For  ATC  to  obtain  the  maximum  benefits 
associated  with  implementing  a  standardized  task  analysis  procedure  and 
documentation  system  and  to  insure  a  rigorous  quality  control  program, 
an  articulated  accountability  system  must  be  defined  and  implemented. 

It  was  recommended  that  a  standardized  task  analysis  procedure 
and  documentation  system  and  improved  procedures  for  in-process  review 
of  task  analysis  efforts  be  developed  and  field  tested  at  ATC  technical 
training  centers.  Further,  it  was  suggested  that  ATC  consider  institut¬ 
ing  both  a  task  analysis  quality  control  program  and  a  fonnal  accounta¬ 
bility  system. 

The  ATC  ISO  Effort 


The  most  pervasive  problem  observed  during  the  site  visits  to  the 
TTCs  was  that  attitudes  toward  ISD  seemed  less  than  optimal,  particu¬ 
larly  at  the  working  level.  In  those  few  instances  where  positive 
attitudes  were  exhibited,  they  were  generally  a  direct  reflection  of 
local  command  attitudes  toward  ISD  or  were  based  on  the  commitment  and 
energy  of  one  or  two  ISD  advocates  at  the  worker  level.  In  reality, 
most  training  development  personnel  had  little  incentive  for  rigorously 
applying  ISD  principles  in  course  design  and  redesign  efforts.  Super¬ 
vision  generally  placed  primary  emphasis  on  compliance  with  ATC  regula¬ 
tions,  and  the  production  and  updating  of  course  control  documents  that 
would  satisfy  inspection  team  criteria.  Standard  operating  procedures 
seemed  invariably  to  dictate  "do  things  as  they  have  always  been  done." 

The  fact  that  ATC  chose  to  diffuse  the  center  level  ISD  teams  and 
to  place  more  responsibility  on  the  branches  at  the  TTCs  seemed  to 
intensify  the  attitudinal  problem.  Perhaps  even  more  important  than 
aggravating  an  already  grave  attitudinal  problem  was  the  fact  that  ISD 
was  to  take  place  exclusively  within  the  context  of  an  operational 
training  system.  The  reality  seemed  to  be  that  day-to-day  training 
operations  took  priority  over  ISD  activities.  The  "old"  centralized  ISD 
teams  were  not  always  effective  and  efficient.  However,  the  survey  seems 
to  show  that  whatever  ineffectiveness  and  inefficiency  occurred  resulted 
primarily  from  problems  associated  with  implementation  of  the  central¬ 
ized  ISD  team  concept  rather  than  any  inherent  weakness  in  the  concept 
itself.  There  was  evidence  that  a  dedicated  team  of  SMSs  and  instruc¬ 
tional  designers  could  accomplish  ISD  effectively  and  efficiently. 
Interestingly  the  MAJCOMS,  the  Navy,  industry,  and  universities  are 
successfully  utilizing  the  team  approach. 

The  other  major  problem  noted  during  the  site  visits  was  lack  of 
quality  control  and  the  absence  of  an  accountability  system  for  ISD 
efforts.  Ultimately,  the  group  commander  was  responsible  for  quality 
control  of  ISD  activities  within  the  organization  and  was  directly 
accountable  to  ATC  Headquarters.  A  commonly  expressed  theme  was  that 
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ISD  is  everyone's  responsibility.  However,  unless  all  levels  of  manage¬ 
ment  actively  participated  in  monitoring  ISD  efforts,  and  certain 
individuals  or  organizational  elements  were  held  accountable  for  those 
efforts,  the  chances  of  fully  obtaining  the  benefits  associated  with 
rigorous  application  of  the  ISD  model  would  be  minimal. 

It  was  recommended  that  ATC  create  a  task  force  to  evaluate  current 
ISD  activities.  It  was  suggested  that  particular  emphasis  be  placed  on 
projecting  the  training  and  cost  benefits  associated  with  rigorization 
of  the  ISD  program  using  the  dedicated  team  approach,  and  making  recom¬ 
mendations  regarding  the  nature  of  ATC's  ISD  effort  in  the  1980-1990 
time  frame.  Futher,  an  in-depth  investigation  of  ways  to  optimize  the 
ISD  process  should  be  conducted.  At  a  minimum,  the  investigation  should 
address: 

0  Identification  of  exemplary  ISD  programs  which  could  serve  as 
models. 

0  Cliaracterization  of  current  ISD  teams  in  terms  of  optimum  size 
and  skill  mix. 

0  Recommendations  for  optimizing  team  configuration  for  type 
of  training  (such  as  equipment  versus  nonequipment  oriented) 
and  training  context  (such  as  formal,  correspondence). 

0  Development  of  standards  for  selecting  team  members. 

0  Definition  of  an  integrated  personnel  acquisition  and  training 
program. 

0  Definition  of  an  integrated  quality  control  program  and  a 
definitive  accountability  system. 


There  were  strong  indications  that  some  major  training  problems 
existed.  Taken  together,  the  survey  results  and  the  cursory  review  of 
ISD  courses  suggested  that  current  ISD  and  task  analysis  training  were 
of  questionable  value  for  working  level  training  development  personnel. 
Survey  comments  provided  little  insight  into  the  true  nature  of  the 
problems.  For  example,  comments  heard  most  frequently  were  that  the 
3AIR75130-X  course  "contains  too  much  information,"  "contains  too  little 
information,"  "contains  too  little  detailed  information,"  ‘‘is  more 
appropriate  for  managers  than  for  workers."  As  far  as  working  level 
personnel  were  concerned,  the  courses  were  at  once  too  detailed  and  too 
general.  They  were,  however,  unanimous  in  their  opinion  that  the 
training  would  prove  beneficial  for  others. 


As  previously  noted,  the  course  reviews  indicated  a  number  of 
serious  deficiencies.  Specifically,  the  3AIR75100  course  omitted 
important  aspects  of  the  ISD  model  and  did  not  adequately  address  the 
interactional  nature  of  the  model's  five  steps.  The  3AIR75130-X  course 
was  found  deficient  primarily  as  a  result  of  the  inappropriateness  of 
the  self-paced  nature  of  the  course  and  the  use  of  adjunctive  program¬ 
ming.  These  observations,  coupled  with  the  poor  attitudes  toward  ISD 
and  the  lack  of  standardization  and  sometimes  rigor  in  ISD  and  task 
analysis  practices  evidenced  across  and  within  centers,  suggested  the 
need  for  an  in-depth  evaluation  of  current  ISD  courses.  Special  atten¬ 
tion  should  be  directed  to  those  portions  of  the  courses  that  provide  an 
orientation  to  the  AF  ISD  model  and  indoctrination  in  ATC  policy  regard¬ 
ing  application  of  the  model.  Those  course  sections  devoted  to  task 
analysis  also  seemed  to  warrant  critical  examination. 

ATC  should  be  commended  for  recognizing  the  need  to  provide  formal 
training  in  the  ISD  process  and  its  associated  procedures.  Classroom 
training  alone,  however,  is  not  sufficient  to  insure  an  efficient  ISD 
program.  A  frequently  recurring  theme  in  discussions  with  training 
development  personnel  at  the  centers  was  that  "the  best  way  to  learn  ISD 
is  by  doing  it."  However,  the  survey  showed  a  marked  absence  of  any 
rigorous  on-the-job  training  (OJT)  program  in  the  area  of  ISD.  A 
wel 1 -structured  OJT  program  would  provide  an  appropriate  vehicle  for 
training  development  personnel  to  develop  and  refine  their  ISD  skills. 

It  was  recommended  that  ATC  consider  a  third-party  critical  evalua¬ 
tion  of  current  ISD  training.  At  a  minimum,  the  following  courses 
should  be  examined:  3AIR75100,  Technical  Instructor  Course  (ISD  Unit); 
3AIR75130-X,  Instructional  Systems  Development;  3AZR75133,  Instructional 
System,  Designer;  and  30ST7500-3,  Development  and  Management  of  Instruc¬ 
tional  Systems.  Additionally,  the  feasibility  of  modularizing  the  ISD 
courses  for  the  purpose  of  achieving  differential  levels  of  depth 
within  individual  model  steps  and  across  the  entire  model  should  be 
investigated.  If  modularization  proved  feasible,  a  multiple  track 
instructional  system  could  be  devised  that  would  provide  flexible  ISD 
curricula  tailored  specif ic^il ly  to  the  needs  of  personnel,  their  super¬ 
visors,  and  middle  and  upper  level  managers.  ATC  should  also  consider 
developing  or  procuring  a  task  analysis  training  course  for  immediate 
irnpl  ementation. 

Guidance  Documentation 

ISD  and  Task  Analysis 

During  the  survey,  it  was  noted  that  currently  available  ISD  and 
task  analysis  guidance  documents  (AFM  50-2  and  AFP  50-58)  were  not 
widely  utilized  by  the  training  development  personnel  at  the  centers. 
In  fact,  many  groups  had  great  difficulty  locating  a  copy  of  either 
document.  There  is  a  need  to  examine  the  current  dissemination  policy 
and  local  distribution  procedures  to  insure  that  current  and  anticipated 
documentary  guidance  is  readily  available  to  all  intended  users. 


Frequently  heard  comments  regarding  these  documents  included, 
"too  complex,"  "require  too  much  paperwork,"  and  "most  applicable  to 
the  design  of  new  courses."  A  review  of  these  documents  supports  the 
excessive  procedural  complexity  and  paperwork  comments.  There  seemed  to 
be  a  legitimate  need  for  a  simplified  task  analysis  procedure  and 
documentation  system  that  could  be  applied  in  both  the  development  and 
revision  of  technical  training  courses.  Additionally,  a  preliminary 
assessment  of  the  feasibility  of  implementing  an  automated  storage/ 
retrieval  system  for  task  analysis  data  seemed  warranted.  It  was  felt 
that  this  type  of  data  bank  could  constitute  an  important  element  in  an 
accountability  system  and  would  certainly  have  the  beneficial  effect  of 
improving  institutional  memory. 

One  final  observation  regarding  the  utilization  of  the  information 
sources  relevant  to  task  analysis  was  made.  Most  groups  interviewed 
seemed  to  be  making  less  than  optimal  use  of  the  full  range  of  inform¬ 
ation  potentially  available  to  them,  particularly  the  OS.  Technological 
innovations  at  OMC,  most  notably  the  Current  Task  Inventory  Bank,  would 
help  solve  the  frequently  cited  problem  of  "lack  of  currency."  Addi¬ 
tionally,  the  guidance  documentation  should  emphasize  the  use  of  the  OS 
in  course  revision  efforts,  and  ISO  training  courses  should  be  revised 
to  insure  that  training  development  personnel  are  fully  aware  of  the 
full  range  of  OS  reporting  options  currently  available.  Current  dissem¬ 
ination  policy  and  local  distribution  procedures  deserved  examination 
and  revision  if  necessary  to  insure  that  all  intended  users  have  access 
to  the  OS  and  other  pertinent  information  sources. 

Based  on  the  survey  findings  and  observations,  it  was  recommended 
that  a  simplified  task  analysis  procedure  and  documentation  system, 
including  improved  procedures  for  in-process  review  of  task  analysis 
efforts,  be  developed  and  field  tested  at  ATC  TTCs.  Additionally,  an 
investigation  into  the  feasibility  of  providing  an  automated  storage/ 
retrieval  system  for  task  analysis  data  was  recommended.  The  Air  Force 
Human  Resources  Laboratory  (AFHRL)  and  ATC  directed  the  development  of  a 
prototype  task  analysis  handbook.  Further,  ATC  agreed  to  support  field 
testing  of  the  prototype  handbook  at  the  TTCs. 


SECTION  III 


DESIGN  GOALS 


HANDBOOK  DESIGN  METHODOLOGY 


The  need  for  a  standardized  task  analysis  procedure  and  docu¬ 
mentation  system  for  use  in  AF  technical  training  had  been  clearly 
established.  Therefore,  the  primary  design  goals  were:  (a)  to  develop 
s  impl  e-to-use,  reliable,  valid,  and  time-efficient  and  cost-effective 
task  analysis/documentation  procedures;  and  (b)  to  provide  a  handbook 
that  would  facilitate  implementation  of  those  procedures  by  ATC  training 
development  personnel. 

DESIGN  PROCESS 


Identification  of  Target  Population 

As  a  result  of  a  reorganization  within  ATC,  the  responsibility 
for  completing  and  documenting  task  analysis  shifted  from  ISD  special¬ 
ists  to  SHSs.  Survey  interviews  revealed  that  military  instructors,  in 
the  intermediate  enlisted  grades,  would  be  the  primary  users  of  the 
handbook. 


Establishment  of  Handbook  Design  Goals  and  Approach 

Five  handbook  design  goals  were  formulated  on  the  basis  of  a 
re-examination  of  information  gathered  in  interviews  and  in-depth 
reviews  of  AFM  50-2  and  AFP  50-58.  They  were  as  follows: 

(1)  Present  a  straightforward,  integrated  task  analysis  procedure. 

(2)  Assume  a  training  standard  or  comprehensive  task  list  as  the 
point  of  departure  for  task  Analysis  activities. 

(3)  Minimize  the  amount  of  instruct ■'onal  analysis  required  of  task 
analysts. 

(4)  Simplify  documentation  requirements  to  the  greatest  extent 
possible. 

(5)  Make  periodic  reviews  by  other  SMSs  an  integral  part  of  the 
task  analysis  process. 

The  design  approach  selected  can  best  be  characterized  as  eclectic. 
It  called  for  utilizing,  to  the  maximum  extent  possible,  existing 
technology  insofar  as  it  was  compatible  with  the  current  AF  ISD  model. 

Review  of  AFM  50-2  and  AFP  50-58 


The  task  analysis  procedures  presented  in  AFM  50-2  and  AFP  50-58 
(Volumes  I  and  II)  were  examined  for  possible  inclusion  in  the  hand¬ 
book.  Useful  guidelines  and  information  were  extracted  and  were  incor¬ 
porated  into  the  handbook. 


Review  of  AF  ISP  Training  Course  Materials 

Air  Force  ISD  training  course  materials  were  examined  to  detennine 
if  the  study  guides  and  workbooks  contained  procedures  and  guide¬ 
lines  that  should  be  included  in  the  prototype  handbook.  Specifically, 
materials  from  the  following  courses  were  examined:  3AZR75100,  Techni¬ 
cal  Instructor;  3AIR75130-X,  Instructional  System  Design;  3AZR75133, 
Instructional  System  Designer;  and  3AIR75160,  Instructional  System 
Materials  Development.  Some  useful  procedures  and  guidelines  from  the 
3AZR75133  and  3AIR75160  courses  were  identified  for  inclusion  in  the 
handbook. 


Review  of  Army  and  Navy  Task  Analysis  Procedures 

In  addition  to  the  AF  ISD  manuals  and  training  course  materials, 
current  Army  and  Navy  task  analysis  guidance  documents  v/ere  also  re¬ 
viewed.  The  United  States  Army  Training  and  Doctrine  Command  Circular 
351-4  contained  a  documentation  system  which  appeared  to  meet  the  design 
goal  of  simplicity  and  seemed  capable  of  accommodating  the  full  range  of 
tasks  that  would  be  encountered  in  ATC  resident  technical  training. 

Review  of  CRI  Procedures 


Interviews  with,  MAJCOM  training  development  personnel  indicated 
that  SAC  and  TAC  relied  heavily  on  the  CRI  approach  to  task  analysis. 
Their  success  with  this  approach  for  ski  1 1 -oriented  training  led  to 
the  decision  to  incorporate  some  CRI-type  analytic  techniques  in  the 
handbook. 


Preparation  of  Prototype  Task  Analysis  Handbook 

Following  completion  of  review  and  information-gathering  activi¬ 
ties,  the  initial  version  of  the  task  analysis  handbook  was  prepared. 
In  keeping  with  design  goals  and  criteria,  a  best-mix  of  available  task 
analysis  methodologies  was  presented  in  a  format  and  at  a  reading  level 
suitable  for  ATC  task  analysts. 

The  prototype  handbook  contained  29  pages  of  text,  18  figures,  and 
14  tables  divided  into  four  major  sections.  The  first  section  provided 
a  general  introduction  to  ISD  and  task  analysis,  and  an  overview  of 
handbook  procedures  and  guidelines.  The  second,  third,  and  fourth 
sections  were  devoted  to  the  development  of  preliminary  criterion 
objectives  (PCOs),  the  identification  of  subtasks,  and  the  specification 
of  supporting  skills  and  knowledges,  respectively.  The  iterative 
process  by  which  the  handbook  was  refined  and  eventually  finalized  is 
described  in  the  sections  that  follow. 
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SECTION  IV 


FIELD  TEST  METHODOLOGY 


GENERAL 


The  field  test  of  the  prototype  task  analysis  handbook  was  conduc¬ 
ted  in  two  stages.  Stage  1  consisted  of  preliminary  tryouts,  while 
Stage  2  was  devoted  to  feasibility  testing.  The  purposes  of  the  prelim¬ 
inary  tryouts  and  feasibility  testing  and  the  procedures  used  are 
described  in  the  paragraphs  that  follow.  The  information  generated 
during  field  testing  was  used  in  preparing  the  final  version  of  the  task 
analysis  handbook. 

PRELIMINARY  TRYOUTS 


Preliminary  tryouts  were  accomplished  to  obtain  information  useful 
for  revising  the  prototype  handbook.  The  goal  was  to  develop  an  empiri¬ 
cal  data  base  that  could  be  used  to  identify  required  revisions  and  make 
the  handbook  as  useful  as  possible.  A  potentially  important  by-product 
of  the  preliminary  tryouts  was  a  set  of  task  analysis  exatiiples  directly 
relevant  to  AF  technical  training. 

Test  Sites  and  Procedures 


Keesler,  Edwards,  and  Chanute  AFBs  were  selected  for  preliminary 
tryout  of  the  handbook.  The  courses  and  specialty  areas  utilized  as 
test  beds  at  each  site  were  as  follows: 

Keesler  AFB 


0  3ABR30430,  Radio  Relay  Equipment  Repairman 
0  3ABR73230,  Personnel  Specialist 
Edwards  AFB 

0  STS  316X0T,  Missile  Systems  Analyst  Specialist  and  Missile 
Systems  Analyst  Technician 

0  STS  431X1E,  Jet  Aircraft  Maintenance  Specialist  and  Jet 
Aircraft  Technician  (Over  Two  Engines) 

0  STS  462X0,  Weapons  Mechanic  and  Weapons  Maintenance 
Technician 

Chanute  AFB 

0  3ABR3163UT,  Missile  Systems  Analyst  Specialist 
0  3ABR75133,  Instructional  System  Designer 
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For  each  course,  a  duty  area  was  selected,  and  a  task  performance  item 
and  a  task  knowledge  item  from  that  duty  area  vvere  chosen  for  analysis. 
For  each  course  at  a  site,  two  SMSs  participated  in  the  preliminary 
tryout:  one  served  as  an  analyst,  the  other  as  a  reviewer. 

Analysts  used  the  handbook  procedures  to  analyze  and  document  one 
task  performance  item  and  one  task  knowledge  item.  Task  analysts  were 
encouraged  to  ask  questions,  identify  problems,  and  present  suggestions 
for  improving  the  procedure.  If  the  analyst  failed  to  understand  an 
explanation,  another  wording  or  elaboration  was  given.  Failures  to 
understand  an  example  were  treated  in  the  same  way.  The  problems 
encountered,  explanations  and  additional  information  provided,  sugges¬ 
tions  for  improvement,  typographical  errors,  and  other  kinds  of  diffi¬ 
culties  that  the  analysts  encountered,  v/ere  recorded.  Reviewers  had  two 
tasks  during  preliminary  tryouts.  Their  primary  task,  of  course, 
consisted  of  reviewing  task  analysis  worksheets  and  documentation.  A 
secondary  function  involved  critically  reviewing  the  handbook  in  an 
attempt  to  identify  faulty  wording,  unclear  passages,  inadequate  expla¬ 
nations,  poor  examples,  improper  sequencing,  poor  layout,  typographical 
errors,  and  other  difficulties.  Additionally,  general  suggestions  for 
improving  the  handbook  and  procedures  described  therein  were  solicited. 

Additionally,  each  Technical  Training  Group  (TTG)  at  each  TTC 
designated  a  senior  review  team,  consisting  of  an  educational  specialist 
and  a  senior  SMS,  to  examine  the  handbook.  The  senior  review  teams 
completed  a  free-response  questionnaire  containing  items  related  to  the 
adequacy  and  practicality  of  the  task  analysis  procedure/documentation 
system  described  in  the  handbook,  and  items  related  to  appropriateness 
of  style  and  format. 

FEASIBILITY  TESTING 


The  primary  purpose  of  feasibility  testing  was  to  assess  the 
recommended  task  analysis  procedure  along  the  following  evaluative 
dimensions;  simplicity/usability,  reliability,  validity  and  time- 
efficiency/cost-effectiveness.  A  secondary  purpose  v/as  to  gather 
additional  information  which  could  be  used  in  revising  the  handbook. 

Test  Sites  and  Procedures 


Lackland,  Lowry,  and  Sheppard  AFBs  were  used  for  feasibility 
testing  (formal  evaluation).  The  courses  and  specialty  areas  utilized 
as  test  beds  at  each  site  were  as  follows: 

Lackland  AFB 


0  3ABR30630,  Electronic  Communications  and  Cryptographic 
Equipment  Systems  Repairman 

0  3ALR7343UA,  Equal  Opportunity  and  Treatment 

0  3AER81130,  Security  Specialist 
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0  3ABR64530,  Inventory  Management  Specialist 
0  3AIR75100,  Technical  Instructor 
Sheppard  AFB 


0  3ABR36131,  Cable  Splicing  Installation  and  Maintenance 
Specialist 

0  3ABR51131A,  Programming  Specialist  (Burroughs  Systems) 
0  3ABR90330,  Radiologic  Specialist 

For  each  of  these  courses,  a  duty  area  was  selected,  and  a  task  perfor¬ 
mance  item  and  a  task  knowledge  item  from  that  duty  area  were  chosen  for 
analysis.  For  each  course  at  each  test  site,  four  SMSs,  one  senior  SMS, 
and  one  training  specialist  participated  in  the  feasibility  testing. 
The  pool  of  four  SMSs  was  divided  into  two  two-member  task  analysis 
teams.  On  each  team,  one  SMS  served  as  the  analyst,  the  other  as 
reviewer.  The  senior  SMS  and  the  training  specialist  served  as  a  task 
analysis  evaluation  team. 

Analysts  utilized  the  handbook  to  analyze  and  document  the  task 
performance  item  and  the  task  knowledge  item.  Those  participants 
designated  as  reviewers  participated  in  the  analysis  and  documentation 
activities  in  the  manner  prescribed  in  the  handbook.  The  amount  of  time 
required  by  each  team  to  complete  each  major  step  in  the  analyses  was 
recorded.  Upon  completing  the  analyses,  each  of  the  analysts  and 
reviewers  was  asked  to  complete  the  Handbook  Evaluation  Survey.  This 
survey  sol/icited  opinions  regarding  the  task  analysis  procedures  as  well 
as  format  and  style.  At  Lowry  and  Sheppard  AFBs,  analysts  and  reviewers 
also  completed  the  Innovation  Evaluation  Survey,  a  multiple  scale 
instrument  which  required  comparative  judgments  about  current  task 
analysis  procedures  and  handbook  procedures.  These  instruments  are  more 
fully  described  in  the  next  subsection.  Data  Collection  Instruments. 

Evaluators  then  reviewed  the  completed  task  and  knowledge  analyses. 
Each  analysis  was  assessed  using  accuracy,  completeness,  and  overall 
adequacy  as  criteria.  The  outcome  of  this  assessment  formed  the  basis 
for  evaluating  the  utility  of  the  handbook  in  the  development  of  objec¬ 
tives,  the  preparation  of  tests,  and  the  design  of  instruction.  The 
evaluators  also  judged  the  degree  of  correspondence  between  the  analyses 
produced  by  the  two  analysis  teams.  The  instruments  used  to  collect 
these  data,  the  Task  Analysis  Evaluation  and  Correspondence  Question¬ 
naires,  are  described  in  the  next  subsection. 

Additionally,  each  TTG  at  each  TTC  provided  a  senior  review  team, 
consisting  of  an  educational  specialist  and  a  senior  SMS,  which  examined 
the  handbook,  identified  problems  and  made  suggestions  for  improvement, 
and  completed  the  Handbook  Evaluation  Survey. 
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Data  Collection  Instruments 


The  five  major  data  collection  instruments  utilized  in  field 
testing  the  task  analysis  procedure  and  handbook  were  the: 

0  Handbook  Evaluation  Survey. 

0  Innovation  Evaluation  Survey.^ 

0  Task  Analysis  Evaluation  Questionnaire.^ 

0  Task  Analysis  Correspondence  Questionnaire.^ 

0  Time  Accounting  Form. 

Descriptions  of  these  instruments  are  provided  in  the  paragraphs  that 
follow. 

The  Handbook  Evaluation  Survey  was  administered  to  analysts, 
reviewers,  and  senior  review  teams.  It  consisted  of  42  Likert-type 
items  that  solicited  opinions  regarding  the  task  analysis  procedures 
prescribed  in  the  handbook,  as  well  as  handbook  format  and  style  (see 
Appendix  A).  Specifically,  inquiries  were  made  regarding: 

0  Format 

0  Writing  style  and  organization 

0  Adequacy  of  the  supporting  materials 

0  Validity 

0  Reliability,  and 

0  Time-efficiency  and  cost-effectiveness. 

Additionally,  three  free-response  items  were  included  to  allow  respon¬ 
dents  to  indicate  those  handbook  features  they  liked  best  and  least  and 
to  raise  important  issues  not  directly  addressed  in  the  survey. 

The  Innovation  Evaluation  Survey  was  also  administered  to  analysts, 
reviewers,  and  senior  review  teams.  It  consisted  of  four  scales  that 
required  respondents  to  make  several  different,  but  related,  judgments 
about  current  task  analysis  procedures  and  the  procedures  prescribed  in 
the  handbook  (see  Appendix  A). 


Prior  to  finial ization  of  the  task  analysis  handbook.  Preliminary 
Performance  Requirements  (PPRs)  were  referred  to  as  PCOs.  Therefore, 
the  term  PCO  was  used  in  this  field  test  questionnaire.  Hereafter,  only 
the  term  PPR  will  be  used  in  the  text. 
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0  Scale  1:  For  each  step  in  the  handbook  procedure,  the  respon¬ 
dent  v/as  asked  to  indicate  if  that  step  was  necessary  for  a 
complete  and  accurate  task  analysis  (Step  necessary:  yes/no). 

0  Scale  2:  For  each  step  in  the  handbook  procedure,  the  respon¬ 
dent  was  asked  to  indicate  the  relative  amount  of  time  expended 
on  that  step  under  current  task  analysis  procedures  (Step 
currently  performed:  yes/no;  if  yes,  relative  amount  of  time 
spent:  1 arge/moderate/smal 1 j. 

0  Scale  3:  For  each  step  in  the  handbook  procedure  performed  as 
part  of  the  current  task  analysis  procedure,  the  respondent  was 
asked  to  indicate  if  the  handbook  procedure  would  be  more, 
equally,  or  less  time  consuming  than  the  cu'^rent  procedure. 

0  Scale  4:  For  each  step  in  the  handbook  procedure  performed  as 
part  of  the  current  task  analysis  procedure,  the  respondent  was 
asked  to  indicate  if  the  handbook  procedure  was  better  than, 
about  the  same  as,  or  worse  than  the  current  task  analysis 
procedure. 

The  Handbook  Evaluation  and  Innovation  Evaluation  Survey  data  sets 
were  utilized  as  primary  information  sources  in  evaluating  the  handbook 
and  the  proposed  task  analysis  procedure.  A  subsequent  subsection 
(Evaluative  Criteria)  details  how  these  data  were  utilized  in  formally 
evaluating  the  handbook. 

The  Task  Analysis  Evaluation  Questionnaire  was  administered  to  Task 
Analysis  Evaluation  Teams.  It  consisted  of  two  item  sets  (see  Appendix 
A).  Item  set  1  required  Task  Analysis  Evaluation  teams  to  make  the 
following  judgments  about  task  performance  and  task  knowledge  analyses: 

0  Does  the  PPR  accurately  reflect  the  behavior  described  in  the 
STS  item? 

0  Is  the  subtask  listing  accurate  in  that  it  reflects  the  hiajor 
steps  that  must  be  accomplished  in  perfoaning  the  task? 

0  Is  the  subtask  listing  thorough  and  complete  in  that  it  reflects 
all  of  the  major  steps  that  must  be  accomplished  in  performing 
the  task? 

0  Are  all  relevant  supporting  skills  and  knowledges  identified? 

Item  set  2  required  judgments  regarding  the  overall  quality  of  the 
analyses  their  accuracy  and  the  utility  of  the  task  analysis  docu¬ 
mentation. 

The  Task  Analysis  Correspondence  Questionnaire  was  also  admin¬ 
istered  to  Task  Analysis  Evaluation  Teams.  It  consisted  of  items  that 
required  judgments  regarding  the  degree  of  correspondence  between  task 
performance  analyses  and  between  task  knowledge  analyses  (see  Appendix 
A).  Specifically,  task  analysis  evaluation  teaniS  answered  the  following 
questions : 


0  Does  the  PPR  of  analysis  #1  closely  match  the  PPR  of  analysis  #2? 


0  Is  there  a  high  degree  of  correspondence  between  the  subtasks 
identified  in  analysis  #1  and  those  identified  in  analysis  #2? 

0  Do  the  supporting  skills  and  knowledges  identified  in  analy¬ 
sis  #1  match,  in  all  important  respects,  those  identified  in 
analysis  #2? 

0  Are  analyses  #1  and  #2  essentially  identical  in  all  major 
respects? 

The  Task  Analysis  Evaluation  and  Correspondence  Questionnaire  data 
sets  were  utilized  as  primary  information  sources  in  evaluating  the 
handbook  and  the  proposed  task  analysis  procedure.  The  subsection  which 
follows  (Evaluative  Criteria)  details  how  these  data  were  utilized  in 
formally  evaluating  the  handbook. 

The  Time  Accounting  Form  contained  blocks  for  recording  time 
expended,  by  the  analyst  and  by  the  reviewer,  in  each  task  analysis 
stage  (see  Appendix  A).  A  form  was  completed  for  each  task  performance 
analysis  and  each  task  knowledge  analysis  accomplished  as  part  of  the 
feasibility  test. 


Evaluative  Criteria 


The  simplicity  and  usability  oy  the  handbook  procedures  was 
assessed  by  examining  analyst,  reviewer,  and  senior  review  team  opinions 
regarding  the  readability  of  the  handbook,  the  clarity  of  the  explana¬ 
tions  offered,  the  adequacy  of  examples  included,  and  the  appropriate¬ 
ness  of  the  terminology.  These  data  were  gathered  with  the  Handbook 
Evaluation  and  Innovation  Evaluation  Surveys.  Additionally,  handbook 
readability  was  assessed  in  accordance  with  the  procedures  specified  in 
MIL-M-38784A. 

The  val  idity  of  the  handbook  procedures  was  assessed  by  examining 
the  opinions  of  the  task  analysis  evaluation  teams  with  regard  to:  the 
accuracy  of  each  analysis;  the  completeness  of  each  analysis;  and  the 
overall  adequacy  of  each  analysis  as  a  basis  for  developing  objectives, 
preparing  tests,  and  designing  instruction.  An  overall  rating  of  the 
quality  of  each  analysis  was  also  solicited.  These  data  were  gathered 
with  the  Task  Analysis  Evaluation  Questionnaire. 

The  rel  iabi  1  ity  of  the  handbook  procedures  was  assessed  by  examin¬ 
ing  the  judged  correspondence  between  task  performance  analyses  and  task 
knowledge  analyses  in  each  course  and  the  consistency  of  correspondence 
across  courses.  The  consistency  with  which  the  new  procedures  produced 
high  quality  results  provided  an  additional  index  of  reliability.  These 
data  were  gathered  with  the  Task  Analysis  Correspondence  and  Evaluation 
Questionnaires. 


The  time-efficiency/ cost-effectiveness  of  handbook  procedures  was 
assessed  by  examining  the  time  required  to  cofnplete  analyses  using  these 
procedures.  These  data  were  used  to  derive  an  estimate  of  the  total 
time  required  to  task  analyze  each  of  the  feasibility  test  courses 
in  its  entirety.  The  handbook  time-to-completion  estimates  were  then 
compared  to  similar  estimates  for  baseline  courses  task  analyzed  using 
current  procedures. 
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SECTION  V 


PRELIMINARY  TRYOUTS 


FINDINGS 


Objec  -ive  and  Results 

The  objective  of  the  preliminary  tryout  phase  of  field  testing  was 
to  gather  information  that  could  be  utilized  to  refine  the  handbook 
prior  to  feasibility  testing  (i.e.,  formal  evaluation).  At  each  test 
site«  those  training  development  personnel  who  participated  as  members 
of  task  analysis  and  senior  review  teams  generated  a  sizable  number  of 
suggestions  for  improving  the  handbook.  There  was  a  substantial  overlap 
between  the  suggestions  put  forth  by  the  two  groups  of  participants  at 
each  test  site.  This  close  agreement  presumably  constituted  consensual 
validation  and  provided  sufficient  justification  for  revising  the  hand¬ 
book  in  accordance  with  the  suggestions  that  were  made.  Not  surpris¬ 
ingly,  both  the  number  of  new  and  the  number  of  major  suggestions 
generated  decreased  steadily  from  site  to  site.  Consequently,  it  was 
concluded  that  the  preliminary  tryouts  had  indeed  served  their  primary 
purpose--a  considerable  amount  of  refinement  had  been  accomplished. 

Handbook  Modifications 


Handbook  modifications  made  as  a  result  of  preliminary  tryouts 
involved  revisions  to  the  procedures  contained  in  the  handbook  and 
changes  in  the  format  and  layout  of  the  handbook.  Major  changes  ana 
revisions  are  described  in  the  paragraphs  that  follow.  Minor  revisions 
included  textual  changes  to  clarify  points  that  caused  confusion  or 
tended  to  be  overlooked  by  the  analysts,  and  to  make  certain  sections 
more  readable  and  understandable. 

Modifications  to  Handbook  Procedures  and  Guidel i nes 


A  major  change  was  made  in  handbook  procedures.  In  the  initial 
version  of  the  handbook,  it  was  assumed  that  each  STo/CTS  item  described 
a  single  task.  It  became  apparent  during  preliminary  tryouts  that  many 
STS/CTS  items  actually  encompassed  a  number  of  related  tasks.  Proce¬ 
dures  were  revised  to  require  analysts  to  examine  each  STS/CTS  item  to 
determine  whether  it  was  "singular"  (encompassed  only  one  task)  or 
"global"  (encompassed  more  than  one  task).  Additionally,  procedures 
for  breaking  global  items  into  component  tasks  were  included  in  the 
handbook. 

Another  problem  encountered  with  the  earliest  version  of  the  hand¬ 
book  concerned  the  preparation  of  the  task  performance  and  task  know¬ 
ledge  lists.  The  handbook  recommended  preparation  of  separate  lists. 
Several  field  test  participants  pointed  out  that  the  end  result  would  be 
a  somewhat  random  list  of  performances  and  knowledges.  Therefore,  a 
single  form  was  devised  to  allow  grouping,  by  duty  area,  of  related  task 
performances  and  task  knowledges. 
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Additional  guidelines  for  dealing  with  tasks  that  were  primarily 
procedural  in  nature  were  also  provided.  The  initial  version  of  the 
handbook  classified  tasks  as  procedural  and  nonprocedural.  It  did  not 
consider  that  class  of  tasks  in  which  some  subtasks  have  to  be  performed 
in  a  prescribed  sequence,  v/hile  others  do  not.  In  the  revised  version, 
analysts  were  directed  to  treat  these  types  of  tasks  as  procedural,  and 
then  define  a  fixed  sequence  for  the  nonprocedural  components. 

Procedures  for  analyzing  tasks  that  are  already  accurately  and 
completely  defined  elsewhere  (e.g.,  in  T.O.s)  were  altered.  During  the 
course  of  preliminary  tryouts,  analysts  were  copying  task  breakdown 
information  from  T.O.s.  The  revised  version  of  the  handbook  directed 
analysts  to  reference  the  appropriate  T.O.  and  proceed  with  the  identi¬ 
fication  and  documentation  of  supporting  skills  and  knowledges. 

An  early  version  of  the  handbook  called  for  completing  subtask 
documentation  prior  to  identifying  supporting  skills  and  knowledges.  It 
quickly  became  apparent  that  this  was  inefficient.  Supporting  skills 
and  knowledges  were  to  be  documented  in  a  column  beside  the  correspond¬ 
ing  subtask.  If  a  particular  subtask  had  numerous  supporting  skills  and 
knowledges,  the  space  left  for  recording  skills  and  knowledges  was 
generally  insufficient.  The  revised  procedures  directed  the  analyst  to 
formally  document  each  subtask  and  then  its  associated  supporting 
skills  and  knowledges. 

Modifications  to  Handbook  Format  and  Layout 

Experience  with  early  versions  of  the  handbook  suggested  that 
some  changes  in  format  and  layout  were  desirable.  The  description 
of  the  purpose  and  scope  of  the  handbook  eventually  becaMe  tlie  lead 
aaragra, jii  in  the  introductory  section.  The  placement  of  the  figures 
and  tables  was  also  altered.  Initially,  all  figures  and  tables  were 
placed  on  foldout  pages  at  the  end  of  the  handbook.  Foldout  pages 
were  utilized  to  maintain  textual  continuity  and  to  eliminate  frequent 
page  turning.  Handbook  users  felt  that  the  figures  and  tables  relevant 
to  a  chapter  should  be  placed  at  the  end  of  that  chapter,  ana  that 
change  was  made. 

FEASIBILITY  TESTING 


Characterization  of  Participants 

Personnel  who  participated  in  the  handbook  feasibility  test  may  be 
conveniently  grouped  into  four  major  categories: 

( 1 )  Task  analysts. 

(2)  Task  analysis  reviewers. 

(3)  Task  analysis  evaluation  teams. 

(4)  Senior  review  teams. 
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Biographical  information  relevant  to  each  participant  group  is  summar¬ 
ized  and  discussed  in  the  paragraphs  that  follow. 

Pertinent  information  for  those  who  participated  as  analysts  during 
the  feasibility  test  are  summarized  in  Table  B-1  of  Appendix  B.  Four¬ 
teen  of  the  16  analysts  were  enlisted  persons,  with  the  majority  holding 
the  rank  of  Staff  Sergeant  or  higher.  The  average  number  of  years  in 
the  AF/Federal  Government  Service  was  7.78.  Most  of  the  analysts  were 
Instructors,  and  the  average  time  in  current  position  was  2.32  years. 
Years  of  field  experience  ranged  from  one  to  nine.  In  addition  to 
the  75100  course  (Technical  Instructor),  a  majority  of  analysts  had 
completed  the  75130  in-service  training  course  (Instructional  System 
Design).  These  data  indicate  that  those  participants  who  served  as 
analysts  were  typical  of  the  target  population  for  whom  the  handbook  was 
intended  -  instructors  with  some  classroom  experience  and  minimal  ISO 
training. 

Pertinent  biographical  information  for  those  who  participated  as 
reviewers  during  feasibility  testing  are  summarized  in  Table  B-2  of 
Appendix  B.  Thirteen  of  the  16  reviewers  were  enlisted  persons,  with 
the  majority  holding  the  rank  of  Staff  Sergeant  or  higher.  The  average 
number  of  years  in  the  AF/Federal  Government  Service  was  10.66.  Most  of 
the  reviewers  were  instructors,  and  the  average  time  in  current  position 
was  1.77  years.  Years  of  field  experience  ranged  from  zero  to  20.  In 
addition  to  the  75100  course,  a  majority  of  reviewers  had  completed  the 
75130  course.  These  data  suggest  that  the  reviewer  and  analyst  popula¬ 
tions  did  not  differ  in  any  major  way  and  that  those  participants  who 
served  as  reviewers  were  similar  to  the  enlisted  personnel  who  might  be 
asked  to  participate  as  SHSs  in  future  task  analysis  efforts. 

Pertinent  biographical  information  for  those  who  participated  as 
evaluators  during  feasibility  testing  are  summarized  in  Table  B-3  of 
Appendix  B.  Thirteen  of  the  17  evaluators  were  enlisted  persons,  with 
the  majority  holding  the  rank  of  Staff  Sergeant  or  tiigher.  The  average 
number  of  years  in  the  AF/Federal  Government  Service  was  12.94.  Tiie 
majority  of  evaluators  were  instructors  and  instructor  supervisors  and 
the  average  time  in  current  position  v;as  1.38  years.  Years  of  field 
experience  ranged  from  two  to  eight.  In  addition  to  the  75100  course,  a 
majority  of  reviewers  had  completed  the  75130  course  and  a  few  had 
completed  the  7500-3  course.  More  experienced  evaluators  with  greater 
seniority  would  have  been  preferred,  but  in  general,  the  evaluators  were 
judged  to  be  capable  of  assessing  the  adequacy  of  task  analyses  and  the 
degree  of  correspondence  between  analyses. 

Pertinent  biographical  information  for  those  who  participated  as 
senior  reviewers  during  feasibility  testing  are  summarized  in  Table  B-4 
of  Appendix  B.  Eight  of  the  17  senior  reviewers  were  civilians,  eight 
were  enlisted  persons,  and  one  was  an  officer.  The  average  number  of 
years  in  the  AF/Federal  Government  Service  was  19.  Most  of  the  senior 
reviewers  were  training  specialists  or  instructor  supervisors,  and  the 
average  time  in  current  position  was  4.39  years.  It  sliould  be  notea, 
however,  that  the  value  for  average  time  in  current  position  was  in¬ 
flated  considerably  by  including  two  training  specialists  who  iiao 
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held  their  positions  for  29  and  18  years,  respectively.  If  thes” 
outlying  values  are  deleted,  the  average  time  in  current  position 
is  1.84  years.  Years  of  field  experience  ranged  from  zero  to  24. 
In  addition  to  the  75100  course,  a  majority  of  tlie  senior  reviewers 
had  completed  the  75130  course  and  several  had  completed  7500  series 
courses.  Those  selected  to  participate  as  senior  reviewers  were  well 
qualified  to  serve  in  that  role. 

Handbook  Simplicity  and  Usability 

The  simplicity  and  usability  of  handbook  procedures  was  assessed  by 
examining  the  judgments  of  task  analysts,  reviewers  and  senior  review 
teams  with  regard  to: 

0  The  handbook  format. 

0  Tlie  writing  style  and  organization  of  the  handbook. 

0  The  handbook  procedures,  with  special  emphasis  on  the  inherent 
logic  and  consistency  of  the  procedures. 

0  The  adequacy  of  supporting  materials. 

These  data,  gathered  with  the  Handbook  Evaluation  Survey,  supported  a 
direct  assessment  of  the  simplicity  and  usability  of  the  handbook  and 
the  procedures  contained  therein.  Additionally,  a  MIL-M-38784A  read¬ 
ability  analysis  was  conducted.  Indirect  assessment  of  simplicity  and 
usability  was  obtained  from  the  responses  of  task  analysts  and  senior 
review  teams  to  Scale  4  of  the  Innovation  Evaluation  Survey. 

Direct  Assessment:  Handbook  Evaluation  Survey  Data 

All  analysts,  reviewers,  and  senior  reviewers  completed  the  Hand¬ 
book  Evaluation  Survey.  The  data  for  those  items  that  assessed  the 
s impl  i c i ty/ usab i 1 i ty  of  the  handbook  and  the  procedures  contained 
therein  are  summarized  in  Appendix  C,  Table  C-1.  This  data  set  was 
examined  in  several  different  ways;  and  the  results  of  those  analyses 
are  summarized  and  discussed  in  the  paragraphs  that  follow.  Analyst  an’d 
reviewer  data  were  pooled,  and  the  response  patterns  compared  with  those 
of  senior  reviewers.  Additionally,  analyst  and  reviewer  response 
patterns  were  compared  as  a  function  of  course  orientation  and  exper¬ 
ience  level  of  participants. 

Courses  were  characterized  as  equipment  or  nonequipment  oriented. 
Equipment  oriented  courses  were  those  that  involved  predominantly 
hands-on  activities,  specifically: 

0  3ABR30630,  Electronic  Communications  and  Cryptographic  Equipment 

Systems  Repairman. 
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0  3ABR36131,  Cable  Splicing  Installation  and  Maintenance  Special¬ 
ist. 

0  3ABR90330,  Radiologic  Specialist. 

Nonequi pment  oriented  courses  were  those  that  dealt  with  primarily 
cognitive  activities,  specifically: 

0  3ALR73430A,  Equal  Opportunity  and  Treatment. 

0  3ABR81130,  Security  Specialist. 

0  3ABR64530,  Inventory  Management  Specialist. 

0  3AIR75100,  Technical  Instructor. 

0  3ABR51131A  Programming  Specialist  (Burroughs  Systems). 

Participant  experience  levels  were  defined  on  the  basis  of  years  of 
field  experience.  Median  splits  produced  analyst  and  reviewer  groups 
with  the  following  profiles: 

0  inexperienced  analysts  -  2.13  years  field  experience  (range: 
1  to  3  years) 

0  experienced  analysts  -  7.38  years  field  experience  (range:  6  to 
9  years ) 

0  inexperienced  reviewers  -  2.13  years  field  experience  (range: 
0  to  4  years ),  and 

0  experienced  reviewers  -  8.75  years  field  experience  (range; 
4.50  to  20  years). 

Format  -  Item  14  of  the  Handbook  Evaluation  Survey  dealt  with  the 
desirability  of  numbering  handbook  paragraphs.  Handbook  paragraphs  were 
not  numbered  in  the  feasibility  test  version  of  the  handbook.  Analysts 
and  reviewers  were  neither  strongly  in  favor  of,  nor  strongly  opposed 
to,  paragraph  numbering  (see  Table  C-1  ).  Analysts  from  nonequipment 
oriented  courses  tended  to  be  more  negative  regarding  parayrapii  number¬ 
ing  than  were  analysts  from  equipment  oriented  courses.  Senior  review¬ 
ers  reacted  negatively  to  the  idea  of  paragraph  numbering.  It  was 
concluded  that  there  was  no  need  to  number  handbook  paragraphs. 

Item  22  sought  opinions  regarding  the  desirability  of  placing 
handbook  examples,  figures,  and  tables  in  a  separate  volume.  Eighty- 
eight  percent  of  the  respondents  indicated  that  a  separate  volume  for 
supporting  materials  was  undesirable  (see  Table  C-1).  Analysts  were 
more  strongly  in  favor  of  a  single,  integrated  volume  than  were  review¬ 
ers.  The  single  volume  preference  was  most  striking  for  analysts  from 
equipment  oriented  courses.  In  general,  experienced  respondents  were 
more  strongly  in  favor  of  a  single  volume  than  were  inexperienced 
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respondents.  Senior  reviewers  were  very  strongly  in  favor  of  having 
textual  explanations  and  examples,  tables,  and  figures  in  a  single 
volume. 

Approximately  A4%  of  the  respondents  agreed  that  the  examples, 
figures,  and  tables  should  be  embedded  in  the  text  (see  item  25,  Table 
C-1  ).  The  patterns  of  responding  for  analysts  and  reviewers  were  highly 
similar.  Partitioning  by  course  type  produced  essentially  identical 
response  patterns.  When  the  data  were  partitioned  by  experience  level, 
some  interesting  differences  surfaced.  Inexperienced  analysts  and 
reviewers  were  strongly  in  favor  of  embedding  the  examples,  tables, 
and  figures  in  the  text;  while  experienced  respondents  were  not.  Senior 
reviewers  were  strongly  opposed  to  embedding  supporting  materials  in  the 
text. 


Item  33  sought  opinions  regarding  the  length  of  the  handbook. 
More  than  70"  of  the  respondents  did  not  feel  that  the  handbook  was 
excessively  long  (see  Table  C-1).  Analysts  in  equipment  oriented 
courses  and  reviewers  in  nonequipment  oriented  courses  most  often 
responded  that  the  handbook  was  of  appropriate  length.  When  the  data 
were  partitioned  according  to  experience  level,  the  experienced  analysts 
and  reviewers  indicated  that  the  handbook  ms  of  appropriate  length  more 
frequently  than  did  their  less  experienced  counterparts.  The  majority 
of  senior  reviewers  (73%)  indicated  that  the  handbook  was  of  appropriate 
length. 

Response  patterns  on  items  related  to  handbook  format  indicated, 
for  the  most  part,  that  respondents  felt  that  the  structure  of  the 
handbook  was  adequate.  Changes  that  would  appeal  to  specific  subgroups 
were  identified.  However,  it  was  believed  that  these  types  of  changes 
might  compromise  the  overall  usefulness  of  the  handbook. 

Writing  Style  and  Organization  -  A  substantial  majority  ,of  the  analysts 
and  reviewers  felt  that  the  writing  was  comprehensible  (see  item  30, 
Table  C-1).  Analyst  and  reviewer  response  patterns  v/ere  essentially 
identical.  Partitioning  on  the  basis  of  type  of  course  and  experience 
level  produced  no  noteworthy  differences.  The  response  pattern  of 
senior  reviewers  matched  that  obtained  for  analysts  and  reviewers. 

Handbook  readability  was  assessed  in  accordance  with  MIL-M-38784A 
procedures.  The  overall  grade  level  was  equal  to  10.97,  with  the  reada¬ 
bility  of  the  samples  ranging  from  10.34  to  12.6.  Opinions  were  also 
sought  regarding  the  readability  of  the  handbook  (see  item  38,  Table 
C-1).  A  majority  of  the  analysts  and  reviewers  indicated  that  the 
hanobook  was  readable.  The  response  patterns  of  analysts  and  reviewers 
were  essentially  identical,  with  reviewers  unanimously  agreeing  that 
the  handbook  was  readable.  There  were  no  marked  differences  in  response 
patterns  as  a  function  of  course  type  and  experience  level.  A  majority 
of  senior  reviewers  (75%)  also  indicated  that  the  handbook  was  readable. 

Response  patterns  to  a  third  item  related  to  reading  demands.  Item 
31,  indicated  that  a  large  majority  of  respondents  (85%)  felt  that 
handbook  reading  demands  were  appropriate  (see  Table  C-1).  No  reliable 
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differences  were  found  between  analysts  and  reviewers,  and  there  were  no 
differences  as  a  function  of  type  of  course  or  level  of  experience. 
Those  few  individuals  who  indicated  that  reading  demands  were  inappro¬ 
priate  were  inexperienced  analysts  and  reviewers.  Senior  review  teams 
were  not  as  positive  regarding  the  appropriateness  of  reading  demands  as 
were  analysts  and  reviewers. 

Item  35  of  the  Handbook  Evaluation  Survey  sought  opinions  regarding 
handbook  terminology.  A  substantial  number  of  analysts  and  reviewers 
indicated  that  the  terminology  was  clear  and  unambiguous  (see  Table 
C-1  ).  Analyst  and  reviewer  response  patterns  did  not  differ.  Analysts 
and  reviewers  from  nonequipment  oriented  courses  indicated  greater 
difficulty  with  the  terminology  than  did  their  counterparts  from 
equipment  oriented  courses.  Response  patterns  did  not  differ  as  a 
function  of  experience  level.  Senior  review  teams  were  evenly  divided 
on  the  issue  of  the  clarity  of  handbook  terminology. 

Two  Handbook  Evaluation  Survey  items  addressed  the  clarity  with 
which  handbook  procedures  are  presented.  The  response  patterns  for  Item 
36  indicated  that  approximately  70%  of  the  respondents  felt  that  hand¬ 
book  concepts  are  clearly  presented  (see  Table  C-1).  Analyst  and 
reviewer  response  patterns  were  essentially  identical.  When  the  data 
were  partitioned  on  the  basis  of  course  type  and  experience  level, 
differences  in  response  patterns  were  negligible.  Fifty  percent  of 
the  senior  reviewers  indicated  that  handbook  concepts  are  clearly 
presented. 

A  majority  of  respondents  (66%)  indicated  that  main  points  were 
easily  identified  (see  Item  24,  Table  C-1).  Again,  no  significant 
differences  in  analyst  and  reviewer  response  patterns  emerged,  and 
differences  in  response  patterns  as  a  function  of  course  type  and 
experience  level  were  negligible.  Seventy-five  percent  of  the  senior 
reviewers  indicated  that  main  points  were  easy  to  identify. 

To  summarize  briefly,  response  patterns  on  items  related  to  hand¬ 
book  writing  style  and  organization  indicated  that  respondents  felt  that 
the  writing  was  comprehensible  and  readable,  the  reading  demands  appro¬ 
priate,  and  tile  terminology  clear  and  unambiguous.  An  independent 
readability  analysis  yielded  an  overall  reading  grade  level  of  10.97. 
Additionally,  a  majority  of  respondents  indicated  the  handbook  concepts 
were  clearly  presented  and  that  main  points  were  easily  identified. 

Consistency.  Logic,  and  Documentation  Requirements  -  Ttie  internal 
consistency  of  the  handbook  task  analysis  procedure  was  assessed  with 
two  items.  Item  18  was  directed  at  assessing  whether  or  not  sufficient 
explanation  was  given  for  each  step  to  permit  understanding  of  the 
task  analysis  procedure  (see  Table  C-1).  Once  again,  the  differences 
between  analyst  and  reviewer  response  patterns  were  small.  Respondents 
from  nonequipment  oriented  courses  were  approximately  twice  as  likely 
to  report  difficulties  as  were  respondents  from  equipment  oriented 
courses.  No  dramatic  differences  surfaced  as  a  result  of  experience 
level  partitioning.  Eight  of  the  12  senior  reviewers  agreed  tiiat 
sufficient  explanation  was  provided. 
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Item  34  also  concerned  the  task  analysis  procedures,  but  was  more 
general  in  nature  than  Item  18.  Again,  approximately  70%  of  the  anal¬ 
ysts  and  reviewers  agreed  that  the  procedures  were  organized  in  such  a 
manner  that  they  were  easy  to  understand  and  apply  (see  Table  C-1 ).  No 
differences  between  the  analyst  and  reviewer  response  patterns  surfaced. 
Differences  obtained  for  equipment  versus  nonequipnient  oriented  courses 
were  not  striking,  nor  were  those  obtained  as  a  function  of  experience 
level.  A  majority  of  the  senior  reviewers  agreed  that  the  procedures 
were  organized  in  such  a  way  that  they  are  easy  to  understand  and 
apply. 


Six  Handbook  Evaluation  Survey  items  were  related  to  the  inherent 
logic  of  the  task  analysis  procedure  prescribed  in  the  handbook.  Item 
39  sought  respondent  opinions  regarding  the  appropriateness  of  convert¬ 
ing  STS/CTS  items  into  behavioral  statements.  Approximately  75%  of  the 
analysts  and  reviewers  agreed  that  the  conversion  of  STS/CTS  items  to 
behavioral  statements  was  appropriate  (see  Table  C-1).  There  was  no 
real  difference  in  the  response  patterns  of  analysts  and  reviewers. 
Analysts  from  equipment  oriented  courses  were  more  positive  regarding 
the  conversion  than  were  analysts  from  nonequipment  oriented  courses, 
indicating  that  conversions  may  be  more  difficult  in  nonequipment 
oriented  courses.  When  the  data  were  partitioned  according  to  exper¬ 
ience  level,  no  dominant  response  patterns  emerged.  More  than  90%  of 
the  senior  reviewers  agreed  that  the  conversion  was  appropriate. 

Item  17  required  respondents  to  indicate  agreement  or  disagreement 
with  the  statement  "The  handbook  should  contain  a  short  explanation  of 
how  STSs  and  CTSs  are  developed."  Analyst,  reviewer,  and  combined 
analyst  and  reviewer  response  patterns  were  essentially  identical,  with 
approximately  50%  agreeing  that  a  short  explanation  should  be  provided 
(see  Table  C-1).  In  each  case,  a  slightly  smaller  proportion  disagreed, 
with  no  marked  differences  as  a  function  of  course  type  or  experience 
level.  Sixty-four  percent  of  the  senior  reviewers  agreed  that  the 
handbook  should  contain  a  short  explanation  of  how  STSs  and  CTSs  are 
developed. 

The  response  patterns  on  Item  27  indicate  that  an  overwhelming 
proportion  of  respondents  felt  that  each  step  in  the  handbook  task 
analysis  procedure  leads  logically  to  the  next  step.  Two-thirds  of  the 
senior  reviewers  were  also  in  agreement  (see  Table  C-1). 

Seventy-five  percent  of  the  analysts  and  69%  of  the  reviewers, 
respectively,  agreed  that  each  step  in  the  task  analysis  procedure  would 
lead  to  the  generation  of  the  information  necessary  to  perform  the  next 
step  (see  Item  21,  Table  C-1).  There  were  no  differences  in  response 
patterns  as  a  function  of  course  type  or  experience  level.  Sixty-seven 
percent  of  the  senior  reviewers  agreed  that  each  step  in  the  procedure 
would  generate  the  information  necessary  to  perform  the  next  step. 

Eighty  percent,  67%,  and  58%  of  the  analysts,  reviewers,  and 
senior  reviewers,  respectively,  agreed  that  the  procedures  for  each 
step  in  the  task  analysis  sequence  are  adequate  to  operate  on  the 
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information  generated  in  the  preceding  step  (see  Item  15,  Table  C-1 ). 
Partitioning  analyst  and  reviewer  data  by  course  type  and  by  exper¬ 
ience  level  produced  some  small,  but  un interpretable,  response  pattern 
differences. 

A  majority  of  the  analysts  and  reviewers  (56%)  reported  that  the 
procedure  was  sufficiently  flexible  (see  Item  5,  Table  C-1).  There  were 
no  differences  between  the  response  patterns  of  analysts  and  reviewers. 
When  the  data  were  partitioned  by  type  of  course,  no  real  differences 
were  obtained,  although  reviewers  from  nonequipment  oriented  courses 
were  less  positive  than  the  other  respondent  groups.  When  the  data  were 
partitioned  according  to  experience  level,  inexperienced  analysts  and 
reviewers  indicated  that  the  procedure  was  sufficiently  flexible  more 
often  than  experienced  analysts  and  reviewers.  A  majority  of  senior 
reviewers  (73%)  indicated  that  the  procedure  was  sufficiently  flexible. 

We  also  sought  opinions  regarding  documentation  requirements. 
Fifty-six  percent  of  the  analysts  and  reviewers  agreed  that  task  dia¬ 
grams  were  necessary  (see  Item  41,  Table  C-1).  Interestingly,  analysts 
favored  task  diagramming  more  strongly  than  reviewers.  Analysts  and 
reviewers  from  equipment  oriented  courses  were  favorably  disposed  toward 
the  diagrams,  as  were  analysts  from  nonequipment  oriented  courses. 
Reviewers  from  nonequipment  oriented  courses,  however,  questioned  the 
need  for  task  diagrams.  Inexperienced  reviewers  and  senior  review 
teams  were  also  unconvinced  of  the  necessity  for  task  diagramming.  Most 
importantly,  however,  those  charged  with  doing  and  documenting  the 
analyses  seemed  convinced  of  the  necessity  of  preparing  task  diagrams. 

A  related  item  (number  19)  called  for  respondent  opinions  regard¬ 
ing  documentation  requirements.  Approximately  half  the  analysts  and 
reviewers  did  not  feel  that  documentation  requirements  were  excessive 
(see  Table  C-1).  Again,  analysts  were  less  likely  than  reviewers  to 
think  that  the  documentation  requirements  were  excessive.  There  was 
little  difference  in  response  patterns  for  analysts  and  reviewers 
from  equipment  and  nonequipment  oriented  courses.  Inexperienced 
analysts  indicated  that  documentation  requirements  were  reasonable 
with  greater  frequency  than  experienced  analysts.  While  the  response 
patterns  for  experienced  and  inexperienced  reviewers  were  essentially 
identical,  senior  reviewers  indicated  that  documentation  requirements 
were  excessive. 

In  summary,  most  respondents  agreed  that  the  handbook  contained 
sufficient  information  and  v/as  organized  in  such  a  way  that  it  was  easy 
to  understand  and  apply  the  task  analysis  procedure.  The  respondent 
sample  was  almost  evenly  divided  on  the  issue  of  modifying  the  handbook 
to  include  a  short  explanation  of  how  STSs  and  CTSs  are  developed.  A 
majority  of  respondents  agreed  that  each  step  in  the  procedure  lead 
logically  to,  and  would  allow  generation  of,  the  information  needed  to 
perform  the  next  step.  Analysts,  reviewers,  and  senior  reviewers 
indicated  that  the  guidelines  for  each  step  in  the  handbook  procedure 
are  adequate  to  operate  on  the  information  generated  in  the  preceding 
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step.  Opinions  regarding  documentation  requirements  were  mixed. 
Analysts,  particularly  inexperienced  analysts,  favored  task  diagramming 
and  indicated  that  handbook  documentation  requirements  were  reasonable. 
Experienced  personnel,  particularly  those  who  simply  reviewed  the 
handbook,  tended  to  express  negative  opinions.  Perhaps  the  handbook 
procedures  require  a  degree  of  rigor  over  and  above  that  required  by 
current  procedures.  Or  it  may  be  that  the  advantages  of  the  proposed 
documentation  only  become  apparent  through  use. 

Adequacy  of  Supporting  Materials  -  Five  Handbook  Evaluation  Survey  items 
were  used  to  el i cit  respondent  opinions  regarding  the  adequacy  of 
supporting  material.  These  items  were  mainly  directed  at  determining  if 
explanations,  tables,  figures,  and  examples  facilitated  the  use  of 
handbook  procedures. 

The  overall  response  pattern  for  Item  28  indicates  that  the 
analysts  and  reviewers  felt  that  the  figures  and  tables  contained  useful 
information.  No  differences  between  the  response  patterns  of  analysts 
and  reviewers  emerged.  Partitioning  on  the  basis  of  course  type  and 
experience  level  produced  no  noteworthy  differences.  A  majority  of 
senior  reviewers  agreed  that  the  figures  and  tables  contained  useful 
information  (see  Table  C-1 ). 

Item  16  assessed  the  adequacy  of  the  explanations  offered  for  the 
steps  of  the  handbook  procedures.  A  majority  of  the  analysts  and 
reviewers  (66%)  indicated  that  the  explanations  were  adequate  (see  Table 
C-1).  There  were  no  differences  between  the  response  patterns  of 
analysts  and  reviewers.  When  response  patterns  were  examined  as  a 
function  of  type  of  course  or  as  a  function  of  experience  level,  no 
significant  differences  surfaced.  Sixty-seven  percent  of  the  senior 
reviewers  agreed  that  explanations  were  sufficiently  comprehensive  to 
permit  use  of  the  handbook  procedures. 

Three  items  were  used  to  determine  if  the  examples  given  in  the 
handbook  were  adequate.  Item  42  was  directed  at  determining  if  the 
handbook  contained  an  adequate  number  and  range  of  examples.  Approxi¬ 
mately  half  the  analysts  and  reviewers  agreed  that  the  number  and  range 
of  examples  were  adequate  (see  Table  C-1).  Analysts  were  less  likely  to 
agree  than  were  reviewers.  When  the  data  were  examined  as  a  function  of 
type  of  course,  analysts  from  nonequipment  oriented  courses  indicated 
that  the  range  and  number  of  examples  was  adequate  more  frequently  than 
did  those  from  equipment  oriented  courses.  Reviewers  from  nonequipment 
oriented  courses  were  also  slightly  more  positive  tlian  reviewers  from 
equipment  oriented  courses.  When  the  data  were  partitioned  according  to 
experience  level,  inexperienced  analysts  were  slightly  more  positive 
than  experienced  analysts.  However,  experienced  reviewers  were  much 
more  likely  to  respond  positively  than  were  inexperienced  reviewers. 
Fifty-eight  percent  of  the  senior  reviewers  felt  that  the  handbook 
provided  an  adequate  number  and  range  of  examples. 
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Respondents  were  asked  to  indicate  if  the  handbook  examples  pro¬ 
vided  adequate  support  for  the  textual  material  (see  Item  13,  Table 
C-1 ).  hore  than  half  of  the  analysts  and  reviewers  indicated  that  the 
examples  were  sufficiently  supportive  of  the  textual  explanations.  An 
examination  of  the  analyst  and  reviewer  data  suggested  that  analysts 
were,  on  the  whole,  more  positive  in  their  opinions  than  were  reviewers. 
When  the  data  were  examined  as  a  function  of  type  of  course,  analysts 
from  nonequipment  oriented  courses  were  more  positive  than  analysts  from 
equipment  oriented  courses;  while  the  opposite  was  true  for  reviewers. 
There  were  no  noticeable  differences  when  the  data  were  examined  as  a 
function  of  experience  level,  except  that  the  inexperienced  analysts, 
the  target  population,  most  strongly  agreed  that  the  range  and  number  of 
examples  was  sufficient.  Sixty-seven  percent  of  the  senior  reviewers 
indicated  that  handbook  examples  provided  adequate  support  for  textual 
materials. 

Seventy-eight  percent,  75%,  and  75%  of  the  analysts,  reviewers,  and 
senior  reviewers,  respectively,  reported  that  the  examples  provided  were 
realistic  and  meaningful  (see  Item  32,  Table  C-1).  There  was  no  relia¬ 
ble  difference  between  the  overall  patterns  of  responding  for  analysts 
and  reviewers,  nor  were  there  marked  differences  in  response  patterns  as 
a  function  of  type  of  course.  When  the  data  were  partitioned  on  the 
basis  of  experience  level,  however,  experienced  analysts  and  reviewers 
were  more  positive  than  inexperienced  analysts  and  reviewers.  The 
majority  of  senior  reviewers  (75%)  agreed  that  the  examples  provided 
were  realistic  and  meaningful. 

Findings  related  to  the  adequacy  of  supporting  materials  may  be 
summarized  as  follows.  First,  analysts  were  more  favorably  disposed 
toward  handbook  examples  than  were  reviewers  or  senior  reviewers. 
Second,  a  substantial  number  of  analysts  and  reviewers  found  that  the 
examples  provided  adequate  support  for  the  text  and  were  both  realistic 
and  meaningful.  Third,  and  lastly,  respondents  seemed  to  be  indicating 
that  a  larger  number  and  range  of  examples  would  be  desirable. 

Indirect  Assessment:  Innovation  Evaluation  Survey  Data 

Scale  2  of  the  Innovation  Evaluation  Survey  required  respondents 
familiar  with  the  handbook  to  indicate  the  relative  amount  of  time 
expended,  under  current  procedures,  on  each  step  of  the  proposed  proce¬ 
dures.  These  time  expenditure  data  are  summarized  and  discussed  in  a 
subsequent  section  of  this  report.  Scale  4  of  the  Innovation  Evaluation 
Survey  required  respondents  familiar  with  the  handbook  to  evaluate  com¬ 
parable  steps  of  the  proposed  and  current  procedures.  Respondents  were 
asked  to  indicate  which  steps  in  the  proposed  procedure  were  better 
than,  about  the  same  as,  or  worse  than  current  procedures.  These  data 
are  summarized  in  Table  C-2.  It  should  be  noted  that  there  is  consider¬ 
able  fluctuation  in  the  number  of  respondents  across  steps.  The  only 
judgments  examined  v/ere  those  of  respondents  who  indicated,  in  complet¬ 
ing  Scale  2,  that  time  was  spent  under  current  procedures,  on  the  step 
in  question.  Examination  of  these  data  indicates  that  handbook  proce¬ 
dures  are  perceived  to  be: 
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0  Better  than  current  procedures  for  the  following  steps:  3,  4,  5, 

6,  7,  8,  9,  10,  11,  12,  13,  14,  15,  18,  19,  and  20. 

0  As  good  as  current  procedures  for  the  following  steps:  1,  2,  9, 

16,  and  17. 

In  the  opinion  of  this  respondent  sample,  handbook  procedures  represent 
a  considerable  improvement  over  current  procedures. 

Val idity 

The  validity  of  handbook  procedures  was  assessed  by  examining 
the  judgments  of  task  analysis  evaluation  teams  with  regard  to: 

0  the  accuracy  of  task  analyses  at  the  PPR,  subtask,  and  skill/ 
knowledge  levels, 

0  the  overall  accuracy,  completeness,  and  quality  of  the  analyses, 

0  the  utility  of  the  resultant  documentation  for  designing  and 
validating  training  programs. 

These  data,  gathered  with  the  Task  Analysis  Evaluation  Questionnaire, 
constituted  a  direct  assessment  of  the  validity  of  the  proposed 
procedures. 

Indirect  assessments  of  the  validity  of  handbook  procedures  were 
based  on  the  responses  of  the  task  analysis  and  senior  review  teams  to 
selected  items  and  scales  from  the  Handbook  Evaluation  and  Innovation 
Evaluation  Surveys.  Pertinent  Handbook  Evaluation  items  dealt  with  the 
degree  of  analytic  accuracy  that  could  be  anticipated  if  handbook 
procedures  were  employed  and  the  perceived  utility  of  the  resultant 
documentation.  The  pertinent  Innovation  Evaluation  Survey  scale  called 
for  judgments  regarding  the  necessity  of  each  step  in  the  proposed 
procedure. 

Direct  Assessment:  Task  Analysis  Evaluation  Questionnaire  Data 

Task  analysis  evaluation  team  judgments  regarding  the  accuracy  of 
task  analyses  at  the  PPR,  subtask,  and  skill  and  knowledge  levels  are 
summarized  in  Appendix  D,  Table  D-1.  The  evaluation  teams  indicated 
that  the  PPRs  prepared  by  the  analysis  teams  accurately  reflected  the 
behaviors  called  for  in  the  STS/CTS  items  in  75%  of  the  cases.  Sub¬ 
tasks,  and  skills  and  knowledges  listings  were  judged  to  be  accurate  and 
complete  in  46%  (mean  of  the  accuracy  and  completeness  component  means) 
and  47%  of  the  cases,  respectively.  There  were  no  marked  differences  in 
accuracy  patterns  as  a  function  of  type  of  analysis,  i.e.,  task  perform¬ 
ance  (TP)  versus  task  knowledge  (TK),  although  the  judged  accuracy  of 
skills  and  knowledges  listings  for  task  knowledge  analyses  was  somewhat 
low.  When  comparisons  were  made  as  a  function  of  course  type,  i.e., 
equipment  oriented  versus  nonequipment  oriented,  no  major  differences 
in  response  patterns  emerged,  although  subtasks  for  equipment  oriented 
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courses  were  more  accurately  identified  than  subtasks  for  nonequipment 
oriented  courses.  These  data  indicate  that  acceptable  levels  of  task 
analytic  accuracy  can  be  achieved  utilizing  handbook  procedures. 

Judgmental  data  regarding  the  overall  accuracy  and  completeness  of 
the  task  analyses  are  summarized  in  Table  D-2.  The  overall  accuracy  and 
completeness  results  were  entirely  consistent  with  the  PPR,  subtask, 
and  skills/knowledges  results  previously  presented.  In  44%  of  the 
cases,  evaluators  agreed  that  the  analyses  were  accurate  and  complete. 
There  were  no  marked  differences  in  judged  accuracy  or  completeness  of 
analyses  as  a  function  of  type  of  analysis  or  type  of  course  (see  Tables 
D-3  and  D-4). 

Overall  quality  ratings  are  summarized  in  Table  0-5.  It  should  be 
noted  that  78%  of  the  32  analyses  performed  were  considered  to  be  of 
satisfactory  quality,  i.e.,  only  22%  of  the  analyses  received  the  "poor" 
rating.  Approximately  one-third  of  the  analyses  uere  rated  "very  good" 
or  "excellent."  There  were  no  marked  differences  in  judged  quality  of 
analyses  as  a  function  of  type  of  analysis  or  type  of  course  (see  Tables 
D-6  and  D-7  ).  These  quality  data,  in  conjunction  with  the  accuracy  and 
completeness  data,  lend  additional  support  to  the  conclusion  that 
acceptable  levels  of  analytic  accuracy  can  be  achieved  with  handbook 
procedures. 

Data  related  to  the  judged  utility  of  the  task  analysis  documenta¬ 
tion  produced  during  feasibility  testing  are  summarized  in  Table  D-8. 
The  majority  of  evaluators  agreed  that  the  documentation  would  facili¬ 
tate  the  development  of  objectives,  test  items,  and  Parts  I  and  II  of 
the  POI.  Ninety-one  percent  of  the  evaluators  agreed  that  the  task 
analysis  documentation  would  facilitate  interpretation  of  FERs  and  would 
provide  a  good  mechanism  for  insuring  that  only  "need  to  know"  informa¬ 
tion  is  included  in  a  course.  More  importantly,  54%  of  the  respondents 
agreed  that  the  documentation  would  adequately  support  the  development 
of  objectives  and  test  items  by  an  SMS  other  than  the  one  who  had 
completed  the  analysis.  There  were  no  marked  differences  in  judged 
utility  of  documentation  as  a  function  of  type  of  analysis,  TP  versus  TK 
(see  Table  D-9).  Additionally,  there  were  no  marked  differences  in 
judged  utility  of  documentation  as  a  function  of  course  type,  equipment 
oriented  versus  nonequipment  oriented  (see  Table  D-10);  although  a 
greater  percentage  of  respondents  favored  the  equipment  oriented  docu¬ 
mentation  (83%  versus  61%).  These  data,  in  conjunction  with  the  data 
cited  and  discussed  previously,  adequately  attest  to  the  validity  of  the 
handbook  procedures. 

Indirect  Assessment:  Handbook  Evaluation  Survey  Data 

Five  Handbook  Evaluation  Survey  items  bore  directly  on  the  issue  of 
the  validity  of  handbook  procedures,  and  the  pertinent  data  are  sununar- 
ized  in  Table  D-11.  Approximately  90%  of  the  analysts  and  reviewers 
agreed  that  the  documentation  generated  in  applying  handbook  procedures 
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would  be  useful  in  developing  learning  objectives.  No  differences  in 
analyst  and  reviewer  response  patterns  surfaced.  There  were  no  marked 
differences  in  analyst  and  reviewer  response  patterns  as  a  function  of 
type  of  course  or  level  of  experience.  The  response  pattern  for  the 
senior  reviewers  was  essentially  identical  to  that  of  the  analysts  and 
reviewers. 

Thirty-seven  percent  and  50%  of  the  analysts  and  reviewers,  respec¬ 
tively,  agreed  that  the  task  analysis  documentation  produced  in  applying 
procedures  would  be  useful  in  preparing  written  tests  and  performance 
checklists.  There  were  no  marked  differences  in  analyst  and  reviewer 
response  patterns  as  a  function  of  type  of  course.  Experienced  analysts 
and  reviewers  agreed  that  the  documentation  produced  would  be  useful  in 
test  and  checklist  development  with  greater  frequency  than  did  inexperi¬ 
enced  analysts  and  reviewers.  Finally,  senior  reviewers  tended  to  be 
more  positive  with  regard  to  the  utility  of  documentation  than  were 
their  analyst  and  reviewer  counterparts. 

More  than  half  of  the  analysts  and  reviewers  (67%  and  63%,  respec¬ 
tively)  agreed  that  the  documentation  produced  in  applying  handbook 
procedures  would  be  useful  in  interpreting  and  acting  upon  FERs.  When 
the  data  were  examined  as  a  function  of  type  of  course,  analysts  and 
reviewers  from  equipment  oriented  courses  expressed  stronger  positive 
opinions  than  those  from  nonequipment  oriented  courses.  Response 
patterns  did  not  differ  as  a  function  of  experience  level,  except  that 
experienced  analysts  tended  to  be  more  positive  than  inexperienced 
analysts.  Fifty-eight  percent  of  the  senior  reviewers  agreed  that  the 
task  analysis  documentation  produced  would  facilitate  interpretation  of 
FERs. 

Forty-four  percent  and  54%  of  the  analysts  and  reviewers,  respec¬ 
tively,  agreed  that  the  use  of  handbook  procedures  would  result  in 
a  complete  and  thorough  task  analysis.  It  is  important  to  note  that 
thirty-one  percent  of  the  analysts  and  33%  of  the  reviewers  were  un¬ 
decided.  Only  19%  of  the  respondents  disagreed  with  the  statement 
that  use  of  handbook  procedures  would  result  in  a  complete  and  thorough 
task  analysis.  There  were  no  marked  differences  in  response  patterns 
as  a  function  of  type  of  course,  although  analysts  in  nonequipment 
oriented  courses  v;ere  less  positive  than  those  in  equipment  oriented 
courses.  No  marked  differences  eiierged  wlien  tiie  lata  were  partitioned 
according  to  level  of  experience.  Fifty-eight  percent  of  the  senior 
reviewers  agreed  that  the  use  of  handbook  procedures  would  result  in 
complete  and  thorough  task  analyses. 

Seventy-five  percent,  60%,  and  75%  of  the  analysts,  reviewers,  and 
senior  reviewers,  respectively,  agreed  that  the  use  of  handbook  proce¬ 
dures  would  result  in  an  accurate  task  analysis.  There  were  no  marked 
differences  in  analyst  and  reviewer  response  patterns  as  a  function  of 
course  type  or  experience  level. 

Response  patterns  on  Handbook  Evaluation  Survey  items  related  to 
the  validity  of  handbook  procedures  indicated  that  analysts,  reviewers, 
and  senior  reviewers  felt  that;  .. 


0  utilization  of  handbook  procedures  would  result  in  complete, 
thorough,  and  accurate  task  analyses. 

0  The  documentation  generated  in  applying  handbook  procedures 
would  be  useful  in  developing  learning  objectives,  preparing 
written  tests  and  performance  checklists,  and  interpreting  and 
acting  upon  FERs. 

These  results  further  attest  to  the  validity  of  handbook  procedures. 
Indirect  Assessment;  Innovation  Evaluation  Survey  Data 

Scale  1  of  the  Innovation  Evaluation  Survey  required  respondents 
familiar  with  handbook  procedures  (i.e.,  analysts,  reviewers,  and  senior 
reviewers)  to  indicate  which  steps  in  the  handbook  procedure  were 
necessary  for  a  complete  and  accurate  task  analysis.  Those  data  are 
summarized  in  Table  D-12.  All  20  steps  were  considered  necessary  by  a 
majority  of  respondents,  with  the  proportions  of  positive  responses 
ranging  from  .74  to  .93.  These  data  provide  additional  indirect  evi¬ 
dence  that  handbook  procedures  are  valid. 

Reliability 

The  reliability  of  the  handbook  procedures  was  assessed  by  examin¬ 
ing  the  judgments  of  task  analysis  evaluation  teams  with  regard  to 

0  The  correspondence  between  task  analyses  at  the  PPR,  subtask, 
and  ski  11 /knowledge  levels. 

0  The  overall  correspondence  between  analyses. 

These  data,  gathered  with  the  Task  Analysis  Correspondence  Question¬ 
naire,  constitute  a  direct  assessment  of  the  reliability  of  the  proposed 
procedures.  A  second  direct  reliability  assessment  was  achieved  by 
examining  the  analysis  quality  and  documentation  utility  data  obtained 
using  the  Task  Analysis  Evaluation  Questionnaire. 

An  indirect  assessment  of  the  reliability  of  handbook  procedures 
was  based  on  task  analysis  and  senior  review  team  responses  to  selected 
items  from  the  Handbook  Evaluation  Survey.  One  of  these  items  required 
respondent  judgments  regarding  expected  correspondence  between  analyses 
if  two  analysts,  equal  in  subject  matter  expertise  and  v^orkiny  inde¬ 
pendently,  utilized  handbook  procedures.  The  other  required  judg¬ 
ments  regarding  the  consistency  with  which  high  quality  results  would 
be  achieved  if  handbook  procedures  were  applied  across  all  types  of 
courses. 


direct  Assessment:  Task  Analysis  Correspondence  Questionnaire  Data 

Task  analysis  evaluation  team  judgments  regarding  the  correspon¬ 
dence  between  task  analyses  are  summarized  in  Appendix  E,  Table  £-1. 
Matches  at  the  PPR,  subtask,  and  skill/knowledge  levels  were  achieved  in 
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63%,  32%,  and  25%  of  the  cases,  respectively.  Overall  correspondence 
was  achieved  in  31%  of  the  cases.  These  degrees  of  correspondence  are 
modest.  Some  evaluators  apparently  adopted  a  stringent  matching  criter¬ 
ion  which  required  one-to-one  correspondence  in  wording.  When  the 
correspondence  data  were  re-examined  using  a  more  lenient  matching 
criterion  (matching  at  the  meaning,  rather  than  word,  level),  matches  at 
the  PPR,  subtask,  and  skill/knowledge  levels  were  achieved  in  75%,  38%, 
and  25%  of  the  cases,  respectively  (see  Table  E-2).  These  new  values  do 
not,  of  course,  represent  significant  increases  in  judged  correspon¬ 
dence.  Judged  correspondence  at  the  PPR,  subtask,  and  skill/  knowledge 
levels  did  not  differ  markedly  as  a  function  of  type  of  analysis  (TP 
versus  TK).  It  is  interesting  to  note  that  matching  at  the  subtask  and 
ski  1 1 /knowl  edge  levels  was  better  for  nonequipment  than  equipment 
oriented  courses  under  both  scoring  systems  (stringent  and  lenient). 
Taken  together,  these  data  indicate  that  handbook  procedures  .are  mod¬ 
estly  reliable,  in  terms  of  one  of  the  definitions  of  reliability 
utilized  in  this  research. 

Direct  Assessment:  Task  Analysis  Evaluation  Questionnaire  Data 

Data  discussed  in  the  previous  section  (Validity)  indicated  that 
application  of  handbook  procedures  consistently  resulted  in  satisfactory 
analyses  and  useful  documentation.  This  type  of  consistency  seems  to  be 
as  valid  an  indicator  of  the  reliability  of  the  procedures  as  is  the 
degree  of  correspondence  between  analyses  independently  performed. 
These  consistency  data,  along  with  the  correspondence  data  previously 
presented,  indicate  that  handbook  procedures  are  sufficiently  reliable 
for  use  in  instructional  design  activities. 

Indirect  Assessment:  Handbook  Evaluation  Survey  Data 

Two  Handbook  Evaluation  Survey  items  bore  directly  on  the  issue  of 
the  reliability  of  handbook  procedures,  and  the  pertinent  data  are 
summarized  in  Table  E-3  (see  Items  12  and  10).  Approximately  37% 
of  the  analysts  and  reviewers  agreed  that  "two  analysts,  equal  in 
subject  matter  expertise,  utilizing  handbook  procedures  and  working 
independently,  would  produce  essentially  identical  results,"  while  35% 
disagreed.  Analysts  from  equipment  oriented  courses  were  much  more 
likely  to  agree  that  two  analysts  \vould  produce  equivalent  results  than 
were  analysts  from  nonequipment  oriented  courses  (66%  and  20%,  respec¬ 
tively).  A  similar,  but  attenuated,  pattern  v<as  obtained  for  reviewers 
from  equipment  and  nonequipment  oriented  courses.  When  the  data  were 
partitioned  by  experience  level,  there  were  few  differences  between  the 
experienced  analysts  and  reviewers.  Inexperienced  reviewers,  however, 
agreed  more  frequently  than  did  inexperienced  analysts.  Fifty  percent 
of  the  senior  reviewers  indicated  that  two  analysts,  working  indepen¬ 
dently,  would  produce  essentially  the  same  results. 

A  second  Handbook  Evaluation  Survey  item  was  designed  to  elicit 
opinions  regarding  the  statement  "standardized  application  of  handbook 
procedures  across  all  types  of  courses  would  produce  high  quality 
results  consistently."  Fifty-four  percent  of  the  reviewers  agreed, 
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whereas  only  25%  of  the  analysts  agreed.  It  is  important  to  note  that 
44%  and  20%  of  the  analysts  and  reviewers,  respectively,  were  undecided, 
indicating  a  relatively  small  percentage  of  truly  negative  opinions. 
Response  patterns  for  analysts  and  reviewers  as  a  function  of  course 
type  were  not  dramatically  different,  although  reviewers  affiliated  with 
nonequi prnent  oriented  courses  tended  to  be  more  positive  than  dia  the 
other  subgroups.  Inexperienced  reviewers  tended  to  be  more  positive 
than  did  their  experienced  counterparts.  Forty-one  percent  of  the 
senior  reviewers  agreed  with  the  statement,  34%  disagreed. 

The  response  patterns  for  these  two  items  are  particularly  diffi¬ 
cult  to  interpret.  The  relatively  high  proportion  of  "undecided" 
responses  suggests  that  respondents  may  have  been  confused  by  tiie 
wording  of  the  questions.  For  instance,  the  term  "essentially  identical 
results"  (see  Item  12)  can  be  interpreted  in  any  number  of  ways  ranging 
from  one-to-one  correspondence  in  wording  to  functional  equivalence. 
Although  the  latter  interpretation  is  more  in  keeping  with  the  intent  of 
the  question,  the  correspondence  data  indicated  that  it  was  not  unusual 
for  task  analysis  evaluators  to  look  for  word-to-word  matches.  Handbook 
Evaluation  Survey  respondents  may  have  ignored  the  modifier  "essen¬ 
tially"  and  keyed  on  the  modifier  "identical."  Unfortunately,  iteiii  lU 
contained  an  ambiguous  modifier,  "high,"  subject  to  a  wide  range  of 
interpretations.  Additionally,  it  is  entirely  possible  that  both  of 
these  items  required  respondents  to  speculate  beyond  their  experiential 
frame  of  reference.  Together,  the  ambiguity  in  the  wording  of  iteiiis  and 
the  requirement  to  speculate  without  an  appropriate  frame  of  reference 
may  account  for  the  puzzling  response  patterns  that  surfaced. 

Time-Efficiency/Cost-Effect iveness 

The  time-efficiency/cost-effectiveness  of  the  proposed  procedure 
was  assessed  by  comparing  the  time  required  to  complete  task  analyses 
during  feasibility  testing  with  the  time  required  to  complete  task 
analyses  using  current  procedures.  Time  expenditure  data  for  current 
task  analysis  procedures  were  gathered  from  personnel  who  were  involved, 
during  calendar  year  1973,  in  the  revision  of  resident  technical  train¬ 
ing  courses  (hereafter  referred  to  as  baseline  courses).  It  should  be 
noted  that  time  expenditure  data  were  collected  at  the  STS/CTS  item 
level  during  feasibility  testing,  while  data  for  baseline  courses  were 
historical  and  generally  recorded  at  the  block  level.  A  common  index, 
task  analysis  hours  per  POI  hour,  was  defined  to  compare  handbook  and 
current  procedures  in  terms  of  time  expenditure  requirements.  This 
comparative  evaluation  permitted  a  direct  assessment  of  the  time- 
efficiency/cost-effectiveness  of  handbook  procedures. 

Indirect  assessments  of  the  time-efficiency/cost-effectiveness  of 
handbook  procedures  were  based  on  the  responses  of  task  analysis  and 
senior  review  teams  to  selected  Innovation  Evaluation  Survey  scales  and 
Handbook  Evaluation  Survey  items.  The  Innovation  Evaluation  Survey  data 
of  interest  included: 


0  respondent  indications  of  the  amount  of  time  expended,  under 
current  procedures,  on  each  step  of  the  proposed  procedure; 
and 

0  respondent  judgments  regarding  those  steps  currently  performed 
for  which  the  proposed  procedures  would  be  more,  equally,  or 
less  time  consuming  than  current  procedures. 

The  Handbook  Evaluation  Survey  data  relevant  to  the  time-efficiency/ 
cost-effectiveness  issue  were  the  item  groups  that  dealt  with  training 
development  personnel  skill  qualifications,  manpower  issues,  and  imple¬ 
mentation  issues. 

Direct  Assessment:  Time  Expenditure  Data  for  Current  and  Proposed 
Procedures 


Task  analysis  time  expenditure  data  are  sunmarized  in  Appendix  F, 
Table  F-1.  These  data  indicate  very  little  overlap  in  the  task  analysis 
hours/POI  hour  distributions  for  the  two  data  sets,  with  handbook 
procedures  considerably  less  time  consuming  than  current  procedures.  It 
should  be  noted  that  the  STS/CTS  items  analyzed  during  feasibility 
testing  were,  for  the  most  part,  selected  by  the  task  analysis  teams 
themselves.  Collectively,  the  items  were  most  likely  aiiiong  the  least 
complex  in  the  training  standards  and  those  with  which  the  analysts  and 
reviewers  were  most  familiar.  This  certainly  reduced  the  amount  of  time 
expended  on  task  analysis  activities  during  feasibility  testing.  It  is 
difficult  to  believe,  however,  that  these  factors  are  solely  responsible 
for  the  good  showing  made  by  handbook  procedures.  These  data  indicate 
that  handbook  procedures  are  more  time-efficient  and,  therefore,  more 
cost-effective  than  current  procedures. 

Indirect  Assessment:  Innovation  Evaluation  Survey  Data 

Scale  2  of  the  Innovation  Evaluation  Survey  required  respondents 
familiar  with  handbook  procedures  (i.e.,  analysts,  reviewers,  and  senior 
reviewers)  to  indicate  the  relative  amount  of  time  expended  under 
current  procedures  on  each  step  of  the  proposed  procedure.  These  data 
are  summarized  in  Table  F-2.  It  should  be  noted  that  only  14  of  the  27 
respondents  who  completed  Scale  1  (Step  Necessary?)  also  completed  Scale 
2.  In  several  instances,  respondents  indicated  that  they  had  had  no 
previous  experience  in  task  analysis  and,  therefore,  could  not  complete 
Scale  2. 


Time  expenditure  data  were  examined  to  identify  those  steps  in  the 
proposed  procedure  on  which: 

0  A  considerable  amount  of  time  is  expended  under  current  proce¬ 
dures  (a  majority  of  the  responses  were  "Icirge"  ana  "moderate"), 
ana 

0  A  lesser  amount  of  time  is  expended  under  current  procedures  (a 
raajority  of  the  responses  were  "small"  and  "none"). 
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The  results  v/ere  as  follows. 

Steps  in  the  proposed  procedure  to  which  considerable  time  is 
devoted  under  the  current  procedures  are:  2,  3,  5,  6,  7,  8,  9,  10,  16, 
17,  and  20.  Steps  in  the  proposed  procedure  to  whicti  lesser  amounts  of 
time  are  currently  devoted  are:  1,  4,  11,  12,  13,  14,  15,  18,  and  19. 

In  view  of  the  fact  that  not  all  STS/CTS  contain  global  items 
(Step  1),  it  is  not  surprising  that  a  majority  of  responcents  iudi- 
cated  that  little  or  no  time  was  spent  breaking  up  STS/CTS  items.  The 
response  patterns  for  Steps  4,  13,  and  18  deal  with  the  review  of 
behavioral  statements,  subtasks,  and  supporting  skills  and  knowledges, 
respectively.  Evidently,  minimal  amouncs  of  time  are  expended  on  review 
or  quality  control  activities  under  current  procedures,  in  spite  of  the 
fact  that  a  majority  of  respondents  felt  that  periodic  reviews  wc^e 
necessary  to  insure  thorough  and  accurate  task  analyses  (see  Section 
4.2.3,  validity).  The  fact  that  minimal  time  is  spent  on  review  activi¬ 
ties  under  cu- rent  procedures  would  account  for  why  so  little  time  is 
currently  spent  revising  subtask  ana  supporting  skills  and  knowledge 
listings  (Stars  14  and  19). 

The  response  pattern  for  Step  10  indicates  that  less  time  than 
would  be  expected  is  currently  spent  determining  the  types  of  tasks 
reflected  in  PPRs.  Consequently,  little  time  is  expended  in  selection 
of  appropriate  analysis  techniques  (Step  11). 

At  first  glance,  the  response  patterns  for  Steps  12  and  15  are 
puzzling.  All  indications  are  that  a  great  deal  of  time  is  currently 
expended  identifying  and  analyzing  subtasks.  A  closer  examination  of 
Step  12  led  to  the  conclusion  that  respondents  were  indicating  that 
little  time  is  spent  preparing  task  diagrams.  Step  15,  Analyze  Sub¬ 
tasks,  is  given  step  status  in  tbe  proposed  procedure.  In  reality,  it 
is  comprised  of  two  major  activities  -  identify  supporting  skills 
(Step  16)  and  identify  supporting  knowledges  (Step  17).  Indications  are 
that  considerable  amounts  of  time  are  expended  on  these  activities 
within  the  framework  of  current  procedures. 

Scale  3  of  the  Innovation  Evaluation  Survey  required  respondents 
familiar  with  the  handbook  to  indicate,  for  those  steps  of  the  proposed 
procedure  that  are  performed  as  part  of  the  current  procedure,  if  the 
proposed  procedure  would  be  more,  equally,  or  less  time  consuming. 
These  data  are  summarized  in  Table  F-3.  It  should  be  noted  that  there 
is  considerable  fluctuation  in  the  number  of  responderits  across  steps. 
The  only  judgments  included  in  this  data  set  were  those  of  responcents 
who  had  previously  indicated  that  some  time  was  spenc  on  the  step  in 
question  under  current  procedures. 

Examination  of  these  data  indicates  that  handbook  procedures  would  be 

0  Less  time  consuming  than  current  procedures  for  the  following 
Steps:  5,  6,  7,  8,  9,  13,  14,  18,  and  20. 


49 


0  As  equally  time  consuming  as  current  procedures  for  the  follow¬ 
ing  Steps:  1,  2,  3,  4,  10,  11,  15,  16,  17,  and  19. 

0  More  time  consuming  than  current  procedures  for  Step  12, 
Identify  Subtasks  (Task  Diagrams). 

In  the  opinion  of  this  respondent  sample,  handbook  procedures  represent 
a  considerable  'improvement  over  current  procedures  in  terms  of  time 
expenditure  requirements. 

Indirect  Assessment:  Handbook  Evaluation  Survey  Data 

Ten  Handbook  Evaluation  Survey  items  were  related  to  the  issue  of 
the  time-efficiency/cost-effectiveness  of  handbook  procedu'res.  Two 
items  (3  and  8)  were  directed  at  eliciting  information  concerning  user 
skill  requirements;  four  items  (40,  37,  29,  and  26)  were  concerned  with 
manpower  issues,  and  four  items  (20,  23,  7  and  11)  were  concerned  with 
the  implementation  of  the  handbook  within  ATC.  Pertinent  data  are 
summarized  in  Table  F-4. 

User  Skill  Requirements  -  More  than  70%  of  the  respondents  indicated 
that  extensive  ISD  training  would  not  be  required  to  effectively  apply 
handbook  procedures.  The  response  patterns  of  analysts  and  reviewers 
were  essentially  identical,  and'  there  were  no  marked  differences  as  a 
function  type  of  course.  When  analyst  and  reviewer  data  were  parti¬ 
tioned  by  experience  level,  experienced  analysts  and  reviewers  agreed 
that  extensive  ISD  training  would  not  be  required.  Although  inexperi¬ 
enced  reviewers  concurred  in  this  judgment,  inexperienced  analysts  felt 
that  extensive  training  would  be  required.  A  majority  of  tlie  Senior 
Review  teams  (61%)  also  felt  that  extensive  ISD  training  would  be 
required. 

A  majority  of  the  analysts  and  reviewers  (70%)  felt  that  the 
handbook  could  be  used  most  effectively  by  a  skilled  i nstructor  wi th 
extensive  field  experience  (see  Item  8,  Table  F-4).  There  were  no 
real  differences  in  the  response  patterns  of  analysts  and  reviewers. 
Analysts  in  equipment  oriented  courses  were  more  likely  to  agree  that 
extensive  experience  was  required  than  those  in  nonequi pi.ient  oriented 
courses.  'Ttvere  were  no  differences  of  opinion  as  a  function  of  exper¬ 
ience  level.  Senior  reviewers  generally  agreed  that  a  skilled  instruc¬ 
tor  could  use  the  handbook  to  maximum  advantage. 

Although  a  substantial  majority  of  the  respondents  agreed  that  the 
handbook  would  be  most  useful  in  the  hands  of  a  skilled  instructor,  r.iany 
also  indicated  that  the  procedures  could  be  utilized  by  personnel  witli- 
out  extensive  ISD  training.  These  data  suggest  that  handbook  procedures 
could  be  optimally  exercised  by  experienced  users.  They  also  suggest 
that  the  handbook  would  be  useful  to  less  experienced  training  develop- 
i.ient  personnel. 


Manpower  Issues  -  Since  the  handbook  requires  a  cooperative  effort 
between  an  analyst  and  reviewer,  four  Handbook  Evaluation  Survey  items 
addressed  manpower  issues  related  to  handbook  procedures.  A  majority  of 
respondents  (90%)  agreed  that  periodic  SMS  reviews  are  a  good  way  of 
insuring  accuracy  and  completeness.  Response  patterns  did  not  differ  as 
a  function  of  type  of  course  or  experience  level (see  Item  40,  Table 
F-4).  Senior  reviewers  unanimously  agreed  that  periodic  reviews  are  a 
good  my  of  insuring  accuracy  and  completeness. 

A  related  item,  number  37,  required  respondents  to  comment  on  the 
desirability  of  working  as  two-member  teams.  Seventy  percent  of  the 
analysts  and  reviewers  thought  that  the  team  approach  was  sound  (see 
Table  F-4).  Again,  analyst  and  reviewer  response  patterns  were  highly 
similar,  although  analysts  were  slightly  more  positive.  When  the  data 
were  partitioned  between  equipment  and  nonequipment  oriented  courses, 
analysts  from  equipment  oriented  courses  found  the  two-member  team 
approach  more  suitable  than  did  analysts  from  nonequipment  oriented 
courses.  A  similar  response  pattern  was  obtained  for  reviewers.  When 
the  data  were  partitioned  on  the  basis  of  experience  level,  the  experi¬ 
enced  analysts  and  reviewers  were  more  positive  about  the  desirability 
of  two-member  teams  than  were  the  inexperienced  analysts  and  reviewers. 
The  overall  response  pattern  for  the  senior  review  teams  approximated 
that  of  the  analysts  and  reviev/ers. 

Two  items  addressed  the  manpower  requirement  implications  of  the 
proposed  handbook.  Approximately  30%  of  the  respondents  felt  ttiat 
introduction  of  the  handbook  would  not  require  additional  manpov/er  at 
the  branch  level  (see  Item  29,  Table  F-4).  Although  the  analysts  were 
predominantly  uncertain  about  the  increase  in  manpower,  more  than  half 
the  reviewers  indicated  that  additional  manpower  would  be  required. 
When  the  data  were  partitioned  by  course  orientation,  30%  of  the  anal¬ 
ysts  from  equipment  oriented  courses  indicated  that  no  additional 
manpower  would  be  required,  and  50%  were  uncertain.  A  less  striking, 
but  similar  pattern,  was  obtained  for  analysts  from  nonequipment  orient¬ 
ed  courses.  More  than  two-thirds  of  the  reviewers  from  equipment 
oriented  courses  indicated  that  more  manpov;er  would  be  required.  Review¬ 
ers  from  nonequipment  oriented  courses  were  of  a  similar  opinion, 
although  less  strongly  so.  Partitioning  the  data  by  experience  level 
resulted  in  response  patterns  that  largely  replicated  the  consolidated 
pattern.  A  majority  of  the  senior  reviewers  (54%)  indicated  tliat 
adoption  of  the  handbook  v;ould  require  additional  manpov;er. 

Item  26  required  respondents  to  address  the  broader  issue 
of  whether  or  not  the  handbook  would  require  additional  resources. 
Although  most  analysts  and  reviewers  were  undecided,  almost  one-third 
felt  that  minimal  additional  resources  would  be  required  (see  Table 
F-4).  A  comparison  of  analyst  and  reviewer  response  patterns  inuicated 
that  analysts  were  more  likely  tiian  reviewers  to  inaicate  that  the 
handbook  could  be  implemented  with  minimal  additional  resources.  When 
the  data  were  partitioned  by  type  of  course,  analysts  from  equipiuent 
oriented  courses  were  more  positive  than  analysts  from  nonequi pment 
courses  that  institution  of  handbook  procedures  would  not  require 
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additional  resources.  The  converse  was  true  for  reviewers.  There 
were  very  few  differences  between  the  subgroups  when  the  data  were 
partitioned  according  to  experience  level.  Approximately  tvvo-thirds  of 
the  senior  reviewers  indicated  that  implementation  of  the  handbook 
would  require  additional  resources.  It  should  be  noted  that  the  term 
"additional  resources"  was  probably  interpreted  quite  differently 
across  groups  of  respondents,  so  these  data  are  extremely  difficult  to 
interpret. 

A  majority  of  respondents  agreed  that  the  team  approach  to  task 
analysis  espoused  in  the  handbook  was  sound.  While  many  respondents 
expressed  uncertainty  about  the  manpower  or  resource  impacts  of  handbook 
implementation,  a  majority  of  senior  reviewers  indicated  that  current 
manpower  allotments  would  have  to  be  increased  before  handbook  proce¬ 
dures  could  be  implemented.  Our  data  suggest  that  satisfactory  analyses 
can  be  accomplished  and  docuiuented  in  accordance  with  handbook  pro¬ 
cedures  within  current  manning  levels. 

Implementation  Issues  -  The  analysts  and  reviewers  overwhelmingly  agreea 
that  the  handbook  would  be  useful  when  a  course  is  undergoing  major 
revision  (see  Item  20,  Table  F-4  ).  There  were  no  major  differences 
between  the  analyst  and  reviewer  response  patterns.  Since  the  analysts 
and  reviewers  were  so  strongly  in  agreement  regarding  the  value  of  the 
handbook  in  course  revision  situations,  negligible  differences  in 
response  patterns  emerged  v/hen  the  data  were  partitioned  by  course  type 
and  by  level  of  experience.  Senior  reviewers  were  less  positive  than 
the  analysts  and  reviewers  that  the  handbook  would  be  useful  in  the 
revision  of  existing  courses. 

Sixty-six  percent  of  the  analysts  and  reviewers  felt  that  the 
handbook  would  be  useful  in  the  development  of  new  courses  (see  Table 
F-4).  Patterns  of  responding  for  the  analysts  and  reviewers  were 
essentially  identical.  When  the  data  were  partitioned  by  type  of 
course,  analysts  from  equipment  oriented  courses  agreed  more  frequently 
than  analysts  from  nonequipment  oriented  courses.  Reviewers  from 
equipment  oriented  courses  were  less  likely  to  indicate  that  the  hand¬ 
book  would  be  helpful  in  the  development  of  a  new  course  than  were 
reviewers  from  nonequipment  oriented  courses.  There  were  no  marked 
differences  in  response  patterns  between  analysts  when  the  data  were 
partitioned  according  to  experience  level.  On  the  other  hand,  exper¬ 
ienced  reviewers  were  more  likely  to  indicate  that  the  handbook  would  be 
useful  in  developing  a  new  course  than  were  inexperienced  reviewers. 
Fifty-eight  percent  of  the  senior  reviewers  agreed  that  handbook  proce¬ 
dures  v;ould  be  useful  for  developing  new  courses. 

Fewer  than  half  the  analysts  and  reviewers  believed  that  the 
handbook  would  be  applicable  to  the  entire  range  of  ATC  courses 
(see  item  7,  Table  F-4).  The  difference  betv/een  analyst  ano  reviewer 
response  patterns  was  negligible,  although  analysts  tended  to  select 
the  scalar  extremes  more  frequently  than  did  reviewers.  When  the  data 
were  partitioned  by  course  orientation,  analysts  and  reviewers  from 
equipment  oriented  courses  agreed  that  the  handbook  v/ould  be  applicable 


to  the  entire  range  of  ATC  courses.  The  responses  of  analysts  and 
reviewers  from  nonequipment  courses  were  less  positive.  When  the  data 
were  partitioned  by  experience  level,  experienced  analysts  and  reviewers 
were  more  likely  to  indicate  that  the  iiandbook  would  be  applicable  to 
the  full  range  of  ATC  courses  than  were  inexperienced  analysts  and 
reviewers.  Fifty  percent  of  senior  reviewers  indicated  that  the  hand¬ 
book  would  be  applicable  to  the  full  range  of  ATC  courses. 

Eighty  percent  of  the  analysts  and  reviev^ers  indicated  that  the 
handbook  coulu  be  used  as  part  of  an  ISD  OJT  program  (see  Item  11,  Table 
F-4 ).  There  was  no  marked  difference  in  analyst  and  reviewer  response 
patterns.  When  analyst  and  reviewer  data  were  partitioned  by  course 
orientation  and  by  experience  level,  response  patterns  were  essentially 
identical.  Seventy-five  percent  of  the  senior  reviewers  agreed  that  the 
handbook  would  be  useful  as  part  of  an  ISD  OJT  program. 

In  summary,  a  majority  of  respondents  indicated  that  the  handbook 
would  be  useful  in  revising  existing  courses  and  developing  new  ones, 
and  that  it  could  be  used  as  part  of  an  ISD  on-the-job  training  program. 
Fewer  than  half  of  the  analysts  and  reviewers  polled  believed  that 
handbook  procedures  were  applicable  to  the  full  range  of  ATC  courses. 
Once  again,  many  respondents  were  probably  required  to  speculate  on 
courses  outside  their  experience  domain.  Fifty  percent  of  the  senior 
reviewers  indicated  that  handbook  procedures  had  applicability  to  the 
full  range  of  ATC  courses. 


SECTION  VI 


CONCLUSIONS 


CONCLUSIONS  AND  RECOMMENDATIONS 


Field  test  results  indicated  that  the  task  anal^i-is  handbook  and 
the  procedures  contained  therein  met  the  established  dtsign  criteria 
of  simplicity,  validity,  reliability,  and  time-efficiency/cost- 
effectiveness. 

With  regard  to  simplicity  and  usability,  the  ATC  training  develop¬ 
ment  personnel  who  participated  in  the  field  test  agreed  that  the; 

0  Physical  structure  and  length  of  the  handbook  were  acceptable. 

0  Text  was  comprehensibly  written  and  the  reading  demands  were 
appropriate. 

0  Terminology  was,  for  the  most  part,  clear  and  unambiguous. 

0  Handbook  task  analysis  procedure  was  internally  consistent  and 
logical,  and  clearly  presented. 

0  Textual  explanations  and  supporting  information,  including 
examples,  were  adequate  to  allow  application  of  the  procedures. 

0  Examples  included  were  realistic  and  meaningful. 

It  should  be  noted  that  a  significant  minority  of  field  test  partici¬ 
pants,  however,  were  not  favorably  disposed  toward  the  documentation 
system.  The  negative  opinions  generally  surfaced  among  reviewers  and 
senior  reviewers.  More  importantly,  those  who  served  as  analysts 
reacted  positively  to  the  documentation  system.  This  suggests  that  the 
flexibility  and  potency  of  the  documentation  system  becomes  obvious  only 
after  one  has  worked  with  it. 

With  regard  to  validity,  independent  evaluations  (by  ATC  personnel) 
of  task  analyses  performed  using  handbook  procedures  indicated  that: 

0  PPRs,  subtasks,  and  supporting  skills  and  knowledges  were 
accurately  identified. 

0  The  analyses  were  generally  judged  to  be  accurate  and  complete, 
and  of  satisfactory  quality. 

0  The  documentation  produced  would  facilitate  the  development  of 
learning  objectives,  tests,  and  the  POI. 

Analysts,  reviewers,  and  senior  reviewers  were  of  the  opinion  tliat: 
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0  All  steps  in  the  handbook  procedure  were  necessary  for  a  can- 
plete  and  accurate  task  analysis. 

0  Utilization  of  handbook  procedures  would  result  in  complete, 
thorough,  and  accurate  task  analyses. 

They  also  felt  strongly  that  tlie  docunientation  generated  in  applying 
handbook  procedures  would  facilitate  the  completion  of  subsequent  steps 
in  the  ISO  process. 

With  regard  to  reliability,  independent  evaluations  (by  ATC  person¬ 
nel  )  of  the  task  analyses  performed  using  handbook  procedures  indicated 
that: 

0  Modest  degrees  of  correspondence  were  achieved  between  pairs  of 
analyses  at  the  PPR,  subtask,  and  skill  and  knowledge  levels 
when  a  stringent  matching  criterion  was  invoked. 

0  Correspondence  at  the  PPR  and  subtasks  levels  improved  slightly 
with  a  lenient  matching  criterion. 

More  importantly,  evaluative  data  indicated  that  application  of  hand¬ 
book  procedures  consistently  resulted  in  a  satisfactory  analysis  and 
useful  documentation.  This  type  of  consistency  may  be  a  better  indi¬ 
cator  of  the  reliability  of  the  procedures  than  is  degree  of  judged 
correspondence. 

With  regard  to  time-efficiency/cost-effectiveness,  direct  time 
expenditure  comparisons  suggested  that  handbook  procedures  are  less  time 
consuming  than  are  current  procedures.  Additionally,  the  majority  of 
analysts,  reviewers,  and  senior  reviewers  were  of  the  opinion  that  the 
handbook  procedure  would  be  less  time  consuming  than  the  current  proce¬ 
dure.  Collectively,  these  groups  also  agreed  that  the  handbook  was 
appropriate  for  the  intended  user  population  and  that  the  procedure 
contained  therein  was  applicable  to  the  revision  of  existing  courses  and 
the  development  of  new  ones. 

After  the  field  test  data  were  analyzed  and  interpreted,  a  final 
draft  of  the  handbook  was  prepared.  A  task  analysis  handbook  conference 
was  then  held  at  Randolph  AFB,  Texas  with  representatives  of  ATC, 
OMC,  AFHRL,  and  MDAC-St.  Louis  in  attendance.  All  attendees,  except 
those  from  MDAC-St.  Louis,  represented  agencies  which  would  be  directly 
involved  in  the  implementation  of  the  handbook  or  the  ultimate  users 
of  the  products  of  task  analysis  efforts.  Field  test  results  were 
summarized  and  a  group  discussion  followed  the  formal  presentation. 
Several  worthwhile  revision  suggestions  were  made,  and  these  were 
eventually  incorporated  into  the  final  version  of  the  handbook.  The 
Task  Analysis  Handbook  has  been  published  as  an  AFHRL  Technical  Report 
(AFHRL-TR-79-45(II )). 
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RECOMMENDATIONS 


Based  on  the  results  of  this  research,  it  is  recommended  that  ATC 
consider  implementation  of  handbook  procedures  in  support  of  the  devel¬ 
opment  of  new  resident  technical  courses,  and  the  revision  of  existing 
resident  technical  training  courses,  when  those  courses  are  to  undergo 
substantive  revision.  A  substantive  revision  is  defined  as  one  in  v/hich 
10%  or  more  of  the  course  content  would  be  replaced  or  altered  in  some 
major  way. 

It  is  further  recommended  that  ATC  authorize  an  investigation  of 
the  feasibility  of  a  centralized,  computer-supported  task  analysis  data 
bank.  Training  development  personnel  at  the  TTCs  could  enter  and  store 
locally  approved  task  analysis  documentation  in  the  data  bank  using 
interactive  computer  terminals  interfaced  to  a  centrally  located  host. 
Training  development  and  management  personnel  at  the  TTCs  and  ATC 
Headquarters  could  then  interrogate  the  system  and  examine  current  task 
analysis  documentation  for  resident  technical  training  courses.  A 
preliminary  data  bank  design  is  detailed  in  Appendix  G.  This  type  of 
automated  storage  and  retrieval  system  would  provide  a  mechanism  for 
improving  management  control  of,  as  well  as  facilitating  evaluation  of, 
task  analysis  and  training  development  efforts.  It  would  also  serve  to 
establish  institutional  memory,  minimize  duplication  of  effort,  and 
provide  inexperienced  training  development  personnel  with  a  broad  range 
of  examples  of  documented  analyses. 
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jRBCSfiUO  PAQB  BUMb-NOI  flUffD 


Respondent's  Name: 

Respondent's  Phone  No.:  _ _ _ _ 

Technical  Training  Group  Represented  (No.);  _ 

INSTRUCTIONS 

There  is  a  series  of  statements  on  the  follcwing  pages  wnich  solicits  your 
opinions  regarding  the  task  analysis  handbook »  the  procecures  It  outlines,  and 
the  practicality  of  those  procedures  for  uje  In  ATC.  Please  be  as  oojective  as 
possible  is  responding  to  these  Items,  and  check  only  one  alternative  for  each 
Item.  Remember,  we  are  genuinely  interested  in  your  opinions! 

Space  is  also  provided  on  the  last  page  for  you  to  raise  important  issues 
that  you  feel  are  not  adequately  addressed  by  survey  items,  ^dci ti onal ly ,  we 
welcome  your  suggestions  for  fmpravTng  the  handbook. 


1,  The  type  of  task  analysis  documentation  described  in  the  hanabook  would  be 
useful  in  interpreting  and  acting  upon  Field  Evaluation  Reports* 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Oi sagree 


2.  Task  analysis  documentation  produced  using  handbook  procedures  would 
be  minima! l:y  useful  in  the  preparation  of  written  tests  and  performance 
checklists. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


3.  The  handbook  would  not  be  useful  unless  one  had  extensive  training 
in  ISO. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


4.  The  task  analysis  documentation  generated  in  applying  handbook  procedures 
would  be  useful  in  developing  learning  objectives. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


5.  The  procedure  described  is  too  mechanical,  analysts  need  more  flexibility. 

Strongly  Strongly 

Agree  Agree  Undecided  ui sagree  D' sagree 


6*  The  use  of  the  procedures  aescribed  tn  the  hancbock  would  result  conrolete 
and  thorough  tasic  analyses. 


Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


7.  The  procedures  described  would  not  be  applicable  to  the  full  range  of 
courses  taught  by  ATC. 


Strongly 

Agree  Agree 


Undecided 


Disagree 


Strongly 

Disagree 


8.  The  handbook  would  be  most  useful  in  the  hands  of  a  skilled  instrjctor  with 
extensive  field  experience  in  his  specialty. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Di sagree 


9.  The  procedures  described  in  the  handbook,  if  properly  applied,  would  '"esul: 
in  an  accurate  taste  analysis. 


Strongly 

Agree 


Undecided 


3 i sagree 


Strongly 
Di sagree 


10.  Standardited  application  of  nandtocte  procedures  across  aK  types  of  courses 
would  not  produce  high  quality  results  consistently. 


Strongly 

Agree 


Strongly 
Disagree  Disagree 


i;.  The  handbook  would  be  useful  as  part  of  an  ISO  CJT  progran. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


12.  Two  task  analysts,  equal  in  subject  matter  expertise  and  field  experience, 
using  handbook  procedures  and  working  independently,  would  produce  essen¬ 
tially  the  same  results. 


Strongly 

Agree 


Undecided 


D1 sagree 


Strongly 

Disagree 


13.  Handbook  examples  provide  inadeouate  supoort  for  the  textual 
expl anatl ons. 


Strongly 

Agree 


Agree  Undecided 


Di sagree 


Strongly 
01 sagree 


lA.  Handbook  paragraphs  should  be  numoered, 


Strongly 

Agree 


Undecided 


Di sagree 


Strongly 
Di saaree 


15,  The  procedures  for  each  step  in  the  tas’  ''^alysis  seouence  are  acecuate 
to  operate  on  the  infcmation  generated  the  preceding  step. 


Strongly 

Agree 


Unoecided 


St  '•pro  ■  y 
Z ’ sacree 
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16,  Explanations  are  too  superficial  to  pemit  use  of  the  tasfe  anal:/s':s 
procedures. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


17,  The  handbook  should  contain  a  short  explanation  of  how  STSs  and  CTSs 
are  developed. 

Strongly  Strongly 

Agree  Agree  UnoecTdeo  Di sagree  Dt  sagree 


18,  The  explanations  given  for  each  step  are  suffic*ent  to  pemit  understanding 
of  the  task  analysis  procedures. 

Strongly  Strongly 

Agree  Agree  Undecided  sagree  Disagree 


19.  Documentation  requirements  are  e'cessive. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


20.  It  would  make  good  sense  to  do  this  type  of  task  analysis  wnen  a  course  is 
undergoing  a  major  revision. 

Strongly  Strongly 

Agree  Agree  Undecided  Di sagree  Disagree 


-ES-i 
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21.  Each  step  in  the  task  analysis  procedure  provices  the  information  necessary 
to  perform  the  next  step. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


22.  Examples,  figures,  and  tables  should  be  placed  in  a  separate,  companion 
VO 1 ume . 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


23.  The  handbook  task  analysis  procedures  described  would  be  best  applied  in 
developing  new  courses. 

Strongly  Strongly 

Agree  Agree  Undecided  Oi sagree  Di sagree 


24.  Main  points  are  easy  to  identify. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


25,  Examples,  figures,  and  tables  should  be  integrated  'nto  the  text. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 
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26.  These  task  analysis  procedures  could  be  imol emented  :n  this  Branch  with 
mi nimal  addi tional  resources. 

Strongly  Strongly 

Agree  Agree  Undecided  Di sagree  Disagree 


27,  Each  step  in  the  task  analysis  procedure  leads  logically  to  the  next 
step. 


Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


28.  Figures  and  tables  contain  very  little  useful  information. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


29,  Adoption  of  this  task  analysis  approach  in  this  Branch  would  require 
addi tional  manpower. 

Strongly  Strongly 

^gree  Agree  Undecided  Disagree  Disagree 


30.  The  writing  style  is  comprenens ible. 

Strongly  Strongly 

Agree  Agree  Undeciced  Disagree  Disagree 


-r  - 
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31.  Reading  demands  are  excessive. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


32.  The  examples  provided  are  realistic  and  meaningful. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


33.  The  handbooic  is  too  long. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


3A,  Procedures  are  organized  in  such  a  way  that  they  are  easy  to  understand  and 
apply. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


35.  Handbook  terminology  is  ambiguous  and  confusing. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


HES-7 
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35.  Handbook  concepts  are  clearly  presented. 


Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


37.  Working  in  two-man  teams  is  a  waste  of  time  and  money. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


38.  The  task  analysis  handbook  is  readable. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


39.  Converting  STS/CTS  items  into  behavioral  statements  is  inappropriate. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


40.  Periodic  reviews  by  fellow  Subject-Matter  Specialists  during  tne  course  cf 
the  analysis  are  a  good  way  of  insuring  accuracy  and  completeness. 


Strongly 

Strongly 

Agree 

Agree 

Undecided 

Di s agree 

Disagree 

HES- 


CO 


[«r«] 


41,  It  Is  unnecessary  to  prepare  task  diagrams;  one  can  perform  an  adequate 
task  analysis  directly  on  tne  Task  Analysis  Documentation  Form. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 


42.  The  handbook  provides  an  adequate  number  and  range  of  examples. 

Strongly  Strongly 

Agree  Agree  Undecided  Disagree  Disagree 
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INNOVATION  EVALUATION  QUESTIONNAIRE  (lES) 


Respondent's  Name:  _ _ 

Respondent's  Phone  No,: 

Technical  Training  Group  Represented  (No.):  _ 

INSTRUCTIONS 

Enclosure  1  of  this  package  summarizes,  in  qraohic  form,  the  task 
analysis  procedure  that  is  being  field  tested  by  McDonnell  Douglas 
Corporation  for  the  Air  Force  Human  Resources  Laboratory  and  the 
Air  Training  Command  (ATC).  Scales  1 ,  2,  3 ,  and  4  ask  for  your 
judgments  regarding  certain  steps  in  that  procedure.  Be  as  objective 
as  possible  in  your  responses.  We  are  truly  interested  in  your 
opinions. 
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SCALE  1 


Por  each  step,  please  indicate  whether  or  not  that  step  Is  necessary 
for  a  complete  and  accurate  task  analysis. 


Step  Necessary 


Yes 


No 


Steps 

1.  Breakup  Global  STS/CTS  Items 

2.  List  All  Task  Performances  and 
Task  Knowledges 

3.  Convert  to  Behavioral  Statements 

4.  Review  By  SMSs 

5.  Revise  Behavioral  Statements 

6.  Convert  to  Preliminary  Criterion 
Objectives  (PCOs) 

7.  Review  By  SMSs 

8.  Revise  PCOs 

9.  Document  PCOs 

10.  Determine  Type  of  Task  Reflected 
in  PCO 

n.  Select  Analysis  Technique 

12.  Identify  Subtasks 
(Task  Diagrams) 

13.  Review  By  SMSs 


IES-2 
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14.  Revise  Subtasks 


15.  Analyze  Subtasks 

16.  Identify  Supporting  Skills 

17.  Identify  Supporting  Knowledges 

18.  Review  By  SMSs 

19.  Revise  Supporting  Skills  and 
Knowledges 


20.  Document  Task  Analysis 


SCALE  2 


For  each  step,  please  maicate  ^hetner  or  not  that  step  performed  in  your 
organization  as  part  of  ATC's  current  task  analysis  procedure.  For  each  step 


that  is  performed,  please  indicate  how  much  time  is  spent  accomplishing  that 


step. 


Step  Performed 
as  Part  of  Current 
Procedure 


Relative  Amount 
of  T ime  Spent 
on  Step 


Steps  Yes  No 

1.  Breakup  Global  STS/CTS  Items  _  _ 


Large  Moderate  Small 


2.  List  All  Task  Performances 
and  Task  Knowledges 


3.  Convert  to  Behavioral 
Statements 


4.  Review  by  SMSs 

5.  Revise  Behavioral 
Statements 


6.  Convert  to  Preliminary 
Criterion  Objectives  (PCOs) 

7.  Review  by  SMSs 
3.  Revise  PCOs 

9.  Document  PCOs 

10.  Determine  Type  of  Task 
Reflected  in  PCO 

11.  Select  Analysis  Technique 


:£S-4 


76 


w 


step  Performed 
as  Part  of  Current 
Procedure 


SCALE  2 


Steps 

12.  Identify  Subtasks 
(Task  Diagrams) 


Yes 


No 


Relative  Amount 
of  Time  Spent 
on  Step 

Large  Moderate  Small 


13.  Review  by  SMSs  _  _  _  _ 

14.  Revise  Subtasks  _  _  _  _ 

15.  Analyze  Subtasks  _  _  _  _ 

16.  Identify  Supporting  Skills  _  _  _  _ 

17.  Identify  Supporting 

Knowledges  _  _  _  _ 

18.  Review  by  SMSs  _  _  _  _ 

19.  Revise  Supporting  Skills 

and  Knowledges  _  _  _  _ 

20.  Document  Task  Analysis  _  _  _  _ 

,  lES-: 


I 

77 


78 


SCALE  3 


New  Procedures 
More  Less 


Steps 

Time  Consuming 

Time  Consuming 

12. 

Identify  Subtasks 
(Task  Diagrams) 

13. 

Review  by  SMSs 

14. 

Revise  Subtasks 

15. 

Analyze  Subtasks 

16. 

Identify  Supporting 

Skills 

17. 

Identify  Supporting 
Knowledges 

18. 

Review  by  SMSs 

19. 

Revise  Supporting  Skills 
and  Knowledges 

20. 

Document  Task  Analysis 

About  the  Same 


:es-' 


SCALE  4 


Refer  to  your  responses  on  Scale  2.  For  each  step  that  1s  performed  in 
your  organization  as  part  of  ATC's  current  task  analysis  procedure,  think 
about  how  the  handbook  procedures  compare  with  the  way  things  are  currently 
done.  Are  the  handbook  procedures  better,  worse  or  about  the  same  as  current 
procedures? 

Handbook  Procedures 

Steps  Better  Same  Worse 

1.  Breakup  Global  STS/CTS  Items  _  _  _ 

2.  List  All  Task  Performances 

and  Task  Knowledges  _  _  _ 


3.  Convert  to  Behavioral 
Statements 

4.  Review  by  SMSs 

5.  Revise  Behavioral 
Statements 

6.  Convert  to  Preliminary 
Criterion  Objectives  (PCOs) 

7.  Review  by  SMSs 

8.  Revise  PCOs 

9.  Document  PCOs 

10.  Determine  Type  of  Task 
Reflected  in  PCO 

11.  Select  Analysis  Technique 


ISS-8 
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SCALE  4 


Handbook  Procedures 
Better  Same  Worse 

12.  Identify  Subtasks 

(Task  Diagrams)  _ 

13.  Review  by  SMSs 

14.  Revise  Subtasks  _ _ 

15.  Analyze  Subtasks  _ 

16.  Identify  Supporting  Skills  _  _ 

17.  Identify  Supporting 

Know 1  edges  _ 

18.  Review  by  SMSs 

19.  Revise  Supporting  Skills 
and  Knowledges 


20,  Document  Task  Analysis 


:es-9 
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TASK  PERFORMANCE 
TASK  KNOWLEDGE 
TEAM  1 
TEAM  2 


TASK  ANALYSIS  EVALUATION  QUESTIONNAIRE  (TAEQ) 


Respondent's  Name: _ 

Respondent's  Phone  No. : 

Technical  Training  Group  Represented  (No.): 


INSTRUCTIONS 

There  is  a  series  of  statements  on  the  following  pages  which  solicits  your 
opinions  regarding  a  task  (performance)  (knowledge)  analysis  performed  by  Team 
(1)  (2).  In  performing  this  analysis  the  team  used  the  procedures  and  guidelines 
provided  in  a  new  task  analysis  handbook.  When  you  evaluate  the  analysis, 
please  be  as  objective  as  possible. 

The  last  two  questions  ask  you  to  evaluate  the  documentation  system  pre¬ 
scribed  in  the  task  analysis  handbook.  We  would  welcome  any  suggestions  you 
have  for  improving  the  documentation  system. 
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The  subtask  listing  is  thorough  and  cot^lete  -  it  reflects  all  of  the 
major  steps  that  must  be  acconpfished  in  performing  the  task* 

(Circle  one) 

YES  NO 


If  (5)  NO,  explain. 


All  relevant  supporting  skills  and  knowledges  are  identified. 
(Circle  one) 

YES  NO 


If  (7)  NO,  explain 


9.  The  task  analysis,  as  documented,  is  accurate.  (Check  one) 


Strongly  Agree 
Agree 
Undecided 
Disagree 

Strongly  Disagree 


10.  The  task  analysis,  as  documented,  is  thorough  and  complete. 
(Check  one) 

Strongly  Agree 
— — —  Agree 

_  Undecided 

_  Disagree 

_  Strongly  Disagree 


II.  How  would  you  characterize  the  overall  quality  of  these  task  analyses 
(Check  one) 


Excellent 
Very  Good 
Good 
Fair 
Poor 


12.  The  task  analysis  documentation  would  facilitate  developing  oDjectives 
and  test  Items.  (Check  one) 

_  Strongly  Agree 

Agree 

Undecided 

Disagree 

_  Strongly  Disagree 


13.  The  task  analysis  documentation  would  prove  useful  in  preparing  Part 
of  the  Plan  of  Instruction.  (Check  one) 

Strongly  Agree 

_  Agree 

Undecided 

Disagree 

Strongly  Disagree 
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14.  The  task  analysis  documentation  would  facilitate  preparation  of  Part  II 
of  the  Plan  of  Instruction,  (Check  one) 

_  Strongly  Agree 

_  Agree 

____  Undecided 

_  Disagree 

Strongly  Disagree 


15,  A  Subject-Matter  Specialist  other  than  the  analyst  could  utilize  the 
task  analysis  documentation  to  prepare  learning  objectives  and  test 
items,  (Check  one) 

_  Strongly  Agree 

_  Agree 

Undecided 

Disagree 

_  Strongly  Disagree 


16.  This  kind  of  rigorous  task  analysis/documentation  system  is  best 

applied  when  developing  a  new  course  or  making  a  major  revision  in  an 
existing  course.  (Check  one) 

Strongly  Agree 

_  Agree 

Undecided 

_  Disagree 

_  Strongly  Disagree 


17.  Documenting  task  analyses  at  this  level  of  detail  is  a  sound  way  of 
Insuring  that  only  "need  to  know"  information  is  included  in  a  course. 
(Check  one) 

_  Strongly  Agree 

_  Agree 

Undecided 

Disagree 

Strongly  Disagree 
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18.  This  kind  of  task  analysis  documentation  would  prove  useful  in  inter¬ 
preting  and  acting  upon  Field  Evaluation  Reports.  (Check  one) 

Strongly  Agree 

_  Agree 

Undecided 

_  D 1  sag  ree 

_  Strongly  Disagree 


NOTE:  Disregard  the  next  two  questions  if  answered  previously. 

19.  The  best  feature(s)  of  this  documentation  system  is  (are): 

i 

t 


20.  The  worst  feature(s)  of  this  documentation  system  Is  (are): 


TASK  PERFORMANCE 
TASK  KNOWLEDGE 


TASK  ANALYSIS  CORRESPONDENCE  QUESTIONNAIRE  (TACQ) 

Respondent's  Name: 

Respondent's  Phone  No,: 

Technical  Training  Group  Represented  (No,) ; 


INSTRUCTIONS 

There  Is  a  series  of  questions  on  the  following  pages  which  solicits  your 
opinions  regarding  the  correspondence  between  the  two  task  (perfonnance)  (know¬ 
ledge)  analyses.  These  analyses  were  performed  by  two  task  analysis  teams 
(working  independently)  using  the  procedures  and  guidelines  provided  in  a  new 
task  analysis  handbook.  When  evaluating  the  degree  of  correspondence  between 
analyses*  please  be  as  objective  as  possible. 


X.  The  Preliminary  Criterion  Objectives  (PCOs)  of  Task  Analysis  #1  match 
closely  the  PCOs  of  Task  Analysis  ifZ. 

YES  NO 


2.  If  (I)  NO,  note  and  describe  the  nature  of  the  discrepancies. 


3.  There  is  a  high  degree  of  correspondence  between  the  subtasks 

identified  in  Task  Analysis  #1  and  the  subtasks  identified  in  Task 
Analysis  #2. 

YES  NO 


4.  If  (3)  NO,  note  and  describe  the  nature  of  the  discrepancies.  Explain 
major  discrepancies. 


The  supporting  skins  and  knowledges  listed  in  Task  Analysis  #1  match, 
in  all  important  respects,  the  supporting  skills  and  knowledges  listed 
in  Task  Analysis  #2. 


YES 


NO 


If  (5)  NO,  note  and  describe  the  nature  of  the  discrepancies. 


Task  Analysis  #1  and  Task  Analysis  #2  are  essentially  identical  in  all 
major  respects. 


YES 


NO 


'Ac:-: 
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TIME  ACCOUNTING  FORM  (TAF) 


Center: 
Grouo: 
Course : 
Team 


Date: 

Type  of  Analysis: 
TP  =  Task  Per 
TK  Task  Kno\ 


Familiarization 
Stage  1 
Stage  2 
Stage  3 


Analyst  Reviewer 


Total 


APPENDIX  B 

BIOGRAPHICAL  INFORMATION:  FIELD  TEST  PARTICIPANTS 
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Missing  Data 


F/6  5/9 


ll  ^A0a7  710  HCOOfMELL  DOUGLAS  ASTRONAUTICS  CO-ST  LOUIS  HO 

methods  FOR  collecting  AND  ANALYZING  TASK  ANALYSIS  DATA.<U) 
jUL  80  A  J  ESCHENBRENNER»  P  B  DEVRIES  F33615-77-C-0076 

(fiCLASSIFZED _ AFhRL-TR«»79-»5(  I ) _ NL 


TABLE  B-2  BIOGRAPHICAL  INFORMATION;  REVIEWERS 


Missing  Data 


Sliepmo:  TA  EVAL.  TEAMS 


Worked  independently 


APPENDIX  C 

SIMPLICITY/USABILITY  TABLES 
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TABLE  C.1  JUDGMEHTS  REGAROING  HANDBOOK  SINPLICITY/USABILITY 

[Analysts  •  A;  Revlawtrs  ■  R;  Equipment  ■  Eq;  Non-Equipment  ■  NEq; 
Experienced  ■  Ex;  Inexperienced  ■  lEx;  Senior  Review  Team  ■  SR] 

[Strongly  Agree (SA) ;  Agree  (A);  Undecided  (U);  Disagree  (D);  Strongly 
Disagree  (SD):  n  •  number  of  cell  entries] 

[Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative] 
a.  ^ndbook  Evaluation 
I .  Handbook  Format 


Item  14.  Handbook  paragraphs  should  be  numbered. 


n 

SA 

A 

U 

D 

SO 

A  ^  n 

32 

vOO 

.34 

.25 

.31 

.09 

A 

16 

vOO 

.38 

.19 

.25 

.19 

R 

16 

.00 

.31 

.31 

.38 

.00 

Eq 

A  6 

.00 

.50 

.33 

.17 

.00 

Courtt 

R  6 

.00 

.33 

.33 

.33 

.00 

MEq 

A  10 

.00 

.30 

.10 

.30 

.30 

R  10 

.00 

.30 

.30 

.40 

eOO 

lEx 

A  8 

.00 

.50 

.13 

.00 

.38 

txperlencf  level 

R  8 

.00 

.13 

.38 

.50 

.00 

Ex 

A  8 

.00 

.38 

.25 

.38 

.00 

R  8 

.00 

.50 

.25 

.25 

.00 

SR 

11 

.18 

.18 

.18 

.36 

.09 

Item  22.  Examples,  figures,  and  tables  should  not  be  placed  In  a 
separate,  companion  volume. 


n 

SA 

A 

U 

0 

$0 

A  t  R 

32 

.25 

.63 

.03 

.06 

.03 

A 

16 

.38 

.50 

.06 

.06 

.00 

R 

16 

.13 

.75 

.00 

.06 

.06 

Eq 

A 

6 

.33 

.67 

.00 

.00 

.00 

Course 

R 

6 

.00 

.83 

.00 

.17 

.00 

NEq 

A 

10 

.40 

.40 

.10 

.10 

.00 

R 

10 

.20 

.70 

.00 

.00 

.10 

lEx 

A 

e 

.25 

.50 

.13 

,13 

.00 

Experience  Level 

R 

8 

.13 

.75 

.00 

.13 

.00 

Ex 

A 

8 

.50 

.50 

.00 

,00 

.00 

R 

8 

.13 

.75 

.00 

.00 

.13 

SR 

12 

.33 

.58 

.08 

.00 

.00 

100 


\ 
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TABLE  C-1  (.CONTINUED) 


Itan  25.  Examples,  figures,  and  tables  should  be  Integrated  Into 
the  text. 


n 

SA 

A 

U 

D 

SO 

A  ^  R 

32 

.06 

.38 

.06 

.34 

.16 

A 

16 

.06 

.38 

.06 

.25 

.25 

R 

16 

.06 

.38 

.06 

.44 

.06 

Eq 

A 

6 

.00 

.33 

.17 

.33 

.17 

Course 

R 

6 

.00 

.50 

.17 

.17 

.17 

NEq 

A 

10 

.10 

.40 

.00 

.20 

.30 

R 

10 

.10 

.30 

.00 

.60 

.00 

lEx 

A 

8 

.00 

.63 

.00 

.25 

.13 

Experience  Level 

R 

8 

.13 

.50 

.00 

.38 

.00 

Ex 

A 

8 

.13 

.13 

.13 

.25 

.38 

R 

8 

.00 

.25 

.13 

.50 

.13 

SR 

12 

.00 

.25 

.17 

.42 

.17 

Item  33. 

The  handbook 

Is  of 

appropriate  length. 

n 

SA 

A 

U 

0 

SO 

A  +  R 

32 

.09 

.66 

.13 

.09 

.03 

A 

16 

.13 

.56 

.10 

.13 

.00 

R 

16 

.06 

.75 

.06 

.06 

.06 

Eq 

A  6 

.00 

.83 

.17 

.00 

,00 

Course 

R  6 

.00 

.67 

.00 

.17 

.17 

NEq 

A  10 

.20 

.40 

.20 

.20 

.00 

R  10 

.10 

.80 

.10 

.00 

.00 

lEx 

A  8 

.25 

.38 

.13 

.25 

.00 

Experience  Level 

R  8 

.00 

.63 

,13 

.13 

.13 

Ex 

A  8 

.00 

.75 

.25 

.00 

.00 

SR 

R  8 

.13 

.88 

.00 

.00 

.00 

11 

.09 

.64 

.18 

.00 

.09 

TABLE  C-1  (CONTINUED) 


2.  Writing  Style  and  Organization 
I tan  30.  The  writing  style  Is  comprehensible. 


n 

SA 

A 

U 

D 

SO 

A  +  R 

32 

.13 

.72 

.06 

.09 

.00 

A 

16 

.13 

.75 

.06 

.06 

.00 

R 

16 

.13 

.69 

.06 

,13 

.00 

Eq 

A 

6 

.00 

1.00 

.00 

.00 

.00 

Course 

R 

6 

.00 

.67 

,17 

.17 

.00 

NEq 

A 

10 

.20 

.60 

.10 

.10 

.00 

R 

10 

.20 

.70 

.00 

.10 

.00 

I  Ex 

A 

8 

.25 

.50 

.13 

.13 

.00 

Experience  Level 

R 

8 

.00 

.63 

.13 

.25 

,00 

Ex 

A 

8 

.00 

1.00 

.00 

.00 

.00 

R 

8 

.25 

.75 

.00 

.00 

.00 

SR 

12 

.00 

.83 

.00 

.08 

.08 

Item  38. 

The  task  analysis  i 

handbook 

Is  readable. 

n 

SA 

A 

U 

0 

SO 

A  R 

32 

.22 

.69 

.06 

.03 

.00 

A 

16 

.19 

.63 

.13 

.06 

.00 

R 

16 

,25 

.75 

.00 

.00 

.00 

£q 

A 

6 

.17 

.83 

.00 

.00 

.00 

Course 

R 

6 

.17 

.83 

.00 

.00 

.00 

NEq 

A 

10 

.20 

.50 

.20 

.10 

.00 

R 

10 

.30 

.70 

.00 

.00 

.00 

lEx 

A 

8 

.25 

.50 

.25 

.00 

.00 

Experience  Level 

R 

3 

.13 

,88 

.00 

.00 

.00 

Ex 

A 

8 

.13 

.75 

.00 

.13 

.00 

R 

8 

.38 

.63 

.00 

.00 

.00 

SR 

12 

.00 

.75 

.08 

.08 

.08 

TABLE  C-1  (CONTINUED) 


Item  31.  Reading  demands  are  appropriate. 


n 

SA 

A 

U 

0 

so 

A  +  R 

32 

.13 

.72 

.03 

.13 

.00 

A 

U 

.13 

.75 

.00 

.13 

.00 

R 

16 

.13 

.69 

.06 

.13 

.00 

Eq 

A  6 

.00 

1.00 

.00 

.00 

.00 

Course 

R  6 

.00 

.67 

.17 

.17 

.00 

NEq 

A  10 

.20 

.60 

.00 

.20 

.00 

R  10 

.20 

.70 

.00 

.10 

.00 

I  Ex 

A  8 

,25 

.50 

.00 

.25 

.00 

Experience  Level 

R  8 

.00 

.63 

.13 

.25 

,00 

Ex 

A  8 

.00 

..00 

,00 

.00 

.00 

R  8 

.25 

.75 

.00 

.00 

,00 

SR 

12 

.08 

.58 

.17 

.00 

.17 

Item  35.  Handbook  terminology  Is  clear  and  unambiguous. 


n 

SA 

A 

U 

0 

SO 

A  R 

32 

.09 

.63 

.09 

.16 

.03 

A 

16 

.13 

.56 

.13 

.13 

.06 

R 

16 

.06 

.69 

.06 

.19 

.00 

Eq 

A  6 

.00 

,83 

.17 

.00 

.00 

Course 

R  6 

.00 

1.00 

.00 

.00 

.00 

NEq 

A  10 

.20 

.40 

.10 

.20 

.10 

R  10 

.10 

.50 

.10 

.30 

.00 

lEx 

A  8 

,25 

,38 

,13 

,25 

.00 

Experience  Level 

R  8 

.00 

.88 

.00 

.13 

.00 

Ex 

A  8 

.00 

.75 

.13 

.00 

.13 

R  8 

.13 

.50 

.13 

.25 

.00 

SR 

12 

.00 

.50 

.08 

.25 

.17 

103 


TABLE  C-1  (CONTINUED) 


Item  36.  Handbook  concepts  are  clearly  presented. 

n  SA  A  U 


A  +  R 
A 

R 

Course 

NEq 

Experience  Level 

Ex 

SR 


32 

.06 

.63 

.09 

16 

.13 

.56 

.13 

16 

.00 

.69 

.06 

6 

.00 

.83 

.17 

6 

.00 

.67 

.17 

10 

.20 

.40 

.10 

10 

.00 

.70 

.00 

8 

.25 

.38 

.25 

8 

.00 

.63 

,13 

8 

.00 

.75 

.00 

8 

.00 

.75 

.00 

12 

.00 

.50 

.17 

Item  24.  Main  points  are  easy  to  identify 

n  SA  A  U 


A  +  R 

32 

.03 

.63 

.13 

A 

16 

.00 

.69 

.13 

R 

16 

.06 

.56 

.13 

Eq 

A 

6 

.00 

.67 

.33 

Course 

R 

6 

.17 

.67 

.00 

NEq 

A 

10 

.00 

.70 

.00 

R 

10 

.00 

.50 

.20 

I  Ex 

A 

8 

.00 

.75 

.13 

Experience  Level 

R 

8 

.13 

.63 

.00 

Ex 

A 

8 

.00 

.63 

.13 

R 

8 

.00 

.50 

.25 

SR 

12 

.00 

.75 

.00 
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TABLE  C-1  (CONTINUED) 


b.  Procedures  Evaluation 

1*  Consistency »  Lo9lc»  and  Documentation  Requirements 

Item  18,  The  explanations  given  for  each  step  are  sufficient  to 
permit  understanding  of  the  task  analysis  procedures. 


• 

n 

SA 

A 

U 

D 

SD 

A  + 

R 

32 

.03 

.66 

.03 

.28 

.00 

A 

16 

.06 

.63 

,00 

.31 

.00 

R 

16 

.00 

.69 

.06 

,25 

.00 

Eq 

A 

6 

.00 

.83 

,00 

.17 

.00 

Course 

R 

6 

.00 

,83 

.00 

.17 

.00 

NEq 

A 

10 

.10 

.50 

.00 

.40 

.00 

R 

10 

.00 

.60 

.10 

.30 

.00 

lEx 

Ex 

A 

8 

.13 

.50 

.00 

,38 

.00 

Experience  Level 

R 

A 

8 

a 

.00 

.00 

.88 

.75 

.00 

,00 

.13 

.25 

.00 

.00 

R 

8 

.00 

.50 

.13 

.38 

.00 

SR 

12 

.00 

.67 

.00 

.25 

.08 

Item  34.  Procedures  are  organized  In  such  a  way  that  they  are 
easy  to  understand  and  apply. 


n 

SA 

A 

U 

D 

SO 

A  +  R 

32 

.09 

.59 

.00 

.31 

.00 

A 

16 

,06 

.63 

.00 

.31 

.00 

R 

16 

.13 

.56 

.00 

.31 

.00 

Eq 

A 

6 

.00 

.83 

.00 

.17 

.00 

Course 

R 

6 

.17 

.50 

.00 

.33 

,00 

NEq 

A 

10 

.10 

.50 

.00 

.40 

.00 

R 

10 

.10 

.60 

.00 

.30 

.00 

I  Ex 

A 

8 

.13 

.50 

.00 

.38 

.00 

Experience  Level 

R 

3 

.13 

.50 

.00 

.38 

.00 

Ex 

A 

8 

.00 

.75 

.00 

.25 

.00 

R 

8 

.13 

.63 

.00 

.25 

.00 

SR 

12 

.00 

.75 

.00 

,00 

.25 
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TABLE  C-1  (CONTINUED) 


Item  39.  Converting  STS/CTS  items  into  behavioral  statements 
Is  appropriate. 


n 

SA 

A 

U 

0 

SO 

A  + 

R 

32 

.19 

.56 

.13 

.09 

.03 

A 

16 

.31 

.38 

.19 

.06 

.06 

R 

16 

.06 

.75 

.06 

.13 

.00 

Eq 

A 

6 

.33 

.50 

.00 

.00 

.17 

Course 

R 

6 

.00 

.83 

.00 

.17 

.00 

NEq 

A 

10 

.30 

.30 

.30 

.10 

.00 

R 

10 

.10 

.70 

.10 

.10 

.00 

lEx 

A 

8 

.38 

.38 

.25 

.00 

.00 

Experience  Level 

R 

8 

.00 

.75 

.13 

.13 

.00 

Ex 

A 

8 

.25 

.38 

.13 

.13 

.13 

R 

8 

.13 

.75 

.00 

.13 

.00 

SR 

12 

.33 

.58 

.00 

.08 

.00 

Item  17,  The  handbook  should  contain  a  shor:  explanation  of  how 
STSs  and  CTSs  are  developed. 


n 

SA 

A 

U 

0 

SO 

A  +  R 

32 

.09 

.41 

.09 

.38 

.03 

A 

16 

.19 

.31 

.06 

.44 

.00 

R 

16 

.00 

.50 

.13 

.31 

.06 

Eq 

A 

6 

.33 

.00 

.00 

.67 

.00 

Course 

R 

6 

.00 

.67 

.00 

.17 

.17 

NEq 

A 

10 

.10 

.50 

.10 

.30 

.00 

R 

10 

.00 

.40 

.20 

.40 

.00 

I  Ex 

A 

8 

.25 

.13 

.13 

.50 

.00 

Experience  Level 

R 

8 

.00 

.38 

.25 

.38 

.00 

Ex 

A 

8 

.13 

.38 

.00 

.50 

,00 

R 

8 

.00 

.63 

.00 

.25 

.13 

SR 

n 

.00 

.64 

.09 

.27 

.00 
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TABLE  C-1  (CONTINUED) 


Item  27.  Each  step  1n  the  task  analysis  procedure  leads  logically  to 


the  next  step. 

n 

SA 

A 

U 

0 

SO 

A  R 

32 

.06 

.84 

.03 

.06 

.00 

A 

16 

.13 

.81 

.00 

.06 

.00 

R 

16 

.00 

.38 

.06 

.06 

.00 

Eq 

Course 

NEq 

A 

6 

.17 

.83 

.00 

.00 

.00 

R 

6 

.00 

.83 

.00 

.17 

.00 

A 

10 

.10 

.80 

.00 

.10 

.00 

R 

10 

.00 

.90 

.10 

.00 

.00 

lEx 

Experience  Level 

Ex 

A 

a 

.13 

.88 

.00 

.00 

.00 

R 

A 

a 

8 

.00 

,13 

.88 

.75 

.00 

.00 

.13 

.13 

.00 

.00 

R 

8 

.CO 

.88 

.13 

.00 

.00 

SR 

12 

.00 

.67 

.08 

.25 

.00 

Item  21. 

Each  step 

In  the  task 

analysis  procedure  provides 

the  informat 

necessary 

to  perform  the  next 

Step. 

n 

SA 

A 

U 

0 

SO 

A  +  R 

32 

.03 

.69 

.03 

.25 

.00 

A 

16 

.06 

.69 

.06 

.19 

,00 

R 

16 

.00 

.69 

.00 

.31 

.00 

Eq  S 

NEqg 

6 

.00 

.67 

.17 

.17 

.00 

Course 

6 

10 

.00 

.10 

.67 

.70 

.00 

.00 

.33 

.20 

.00 

.00 

10 

.00 

.70 

.00 

.30 

.00 

TC  A 

lEx  0 

8 

.13 

.63 

.13 

.13 

.00 

Experience  Level 

8 

8 

.00 

.00 

.63 

.75 

.00 

.00 

.38 

.25 

.00 

.00 

8 

.00 

.75 

.00 

.25 

.00 

SR 

12 

.00 

.67 

.08 

.25 

.00 
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TABLE  C-1  (CONTINUED) 


Item  15.  The  procedures  for  each  step  In  the  task  analysis  sequence 
are  adequate  to  operate  on  the  Information  generated  In  the 
preceding  step. 


n 

SA 

A 

U 

0 

SO 

A  + 

R 

30 

.10 

.61 

.06 

.19 

.00 

A 

15 

.20 

.60 

.07 

.13 

.00 

R 

15 

.00 

.67 

.07 

.27 

.00 

Eq 

NEq 

A 

6 

.17 

.83 

.00 

.00 

.00 

Course 

R 

A 

6 

9 

.00 

.22 

.50 

.44 

.17 

.22 

.33 

.22 

.00 

.00 

R 

9 

.00 

.78 

.00 

.22 

.00 

lEx 

Ex 

A 

8 

.25 

.63 

.13 

.00 

.00 

Experience  Level 

R 

A 

8 

7 

.00 

.14 

.50 

.57 

.13 

.00 

.38 

.29 

.00 

.00 

R 

7 

.00 

.86 

.00 

.14 

.00 

SR 

12 

.00 

.58 

,17 

.17 

.08 

Item  5. 

The  procedure  described 

Is  sufficiently  flexible. 

n 

SA 

A 

U 

0 

SO 

A  +  R 

32 

.06 

.50 

.22 

.09 

.13 

A 

16 

.13 

.50 

.19 

.06 

.13 

R 

16 

.00 

.50 

.25 

.13 

.13 

NEq  5 

6 

.00 

.67 

.17 

.00 

.17 

Course 

6 

10 

.00 

.20 

.67 

.40 

.00 

.20 

,17 

.10 

.17 

.10 

10 

.00 

.40 

.40 

.10 

.10 

lEx  5 

-5 

8 

.00 

.75 

.13 

.13 

.00 

Experience  Level 

8 

3 

.00 

.25 

.63 

.25 

.38 

.25 

.00 

.00 

.00 

.25 

8 

.00 

.38 

.13 

.25 

.25 

SR 

11 

.09 

.64 

.27 

.00 

.00 
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TABLE  C-1  (CONTINUED) 


Item  41.  It  Is  necessary  to  prepare  taste  dfagrams;  one  cannot  perform  an 

adequate  tasJc  analysis  directly  on  the  Task  Analysis  Documentation 
Form. 


n 

SA 

A 

U 

0 

SO 

A  ^  ft 

32 

.03 

.53 

.16 

.22 

.06 

A 

16 

.06 

.63 

.06 

.19 

.06 

ft 

16 

.00 

.44 

.25 

.25 

.06 

R 

6 

.00 

.67 

.17 

.17 

.00 

Course 

6 

,00 

,67 

.17 

.17 

.00 

NEq* 

10 

.10 

.60 

.00 

.20 

.10 

10 

.00 

.30 

.30 

.30 

.10 

lEx  i 

-5 

8 

.13 

.63 

.00 

.25 

.00 

Experience  Level 

8 

8 

.00 

.00 

.38 

.63 

.25 

.13 

.25 

.13 

.13 

.13 

8 

.00 

.50 

.25 

.25 

.00 

Sft 

11 

.00 

.27 

.27 

.36 

.09 

Item  19.  Documentation  requirements  are  reasonable. 


n 

SA 

A 

U 

0 

SO 

A  ♦  ft 

32 

.03 

.53 

.13 

.22 

.09 

A 

16 

.06 

.56 

.25 

.13 

.00 

ft 

16 

.00 

.50 

.00 

.31 

.17 

Course 

NEq 

A 

6 

.00 

,67 

.33 

.00 

.00 

ft 

A 

6 

10 

.00 

.10 

.50 

.50 

.00 

.20 

.33 

.20 

.17 

.00 

ft 

10 

.00 

.50 

.00 

.30 

.20 

lEx 

Experience  Level 

Ex 

A 

8 

.13 

.63 

.13 

.13 

.00 

ft 

A 

8 

8 

.00 

.00 

.50 

.50 

.00 

.38 

.13 

.13 

.38 

.00 

ft 

8 

.00 

.50 

.00 

.50 

.00 

Sft 

12 

.08 

.25 

.17 

.42 

.08 
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TABLE  C*1  (CONTINUED) 


2.  Adequacy  of  Supportinq  Materials. 

Item  28.  Figures  and  tables  contain  much  useful  information. 


n 

SA 

A 

U 

0 

SO 

A  ♦  R 

32 

.28 

.66 

.00 

.06 

.00 

A 

16 

.31 

.63 

.00 

.06 

.00 

R 

16 

.25 

.69 

.00 

.06 

.00 

Eq 

Course 

NEq 

A 

6 

.33 

.67 

.00 

.00 

.00 

R 

A 

6 

10 

.17 

.30 

.83 

.60 

.00 

.00 

.00 

.10 

.00 

.00 

R 

10 

.30 

.60 

o 

o 

.10 

.00 

lEx 

Experience  Level 

Ex 

A 

8 

.38 

.63 

.00 

.00 

.00 

R 

A 

8 

8 

.13 

.25 

.88 

.63 

.00 

.00 

.00 

.13 

.00 

.00 

R 

8 

.38 

.50 

.00 

.13 

.00 

SR 

12 

.00 

.75 

.08 

.08 

.08 

Item  16.  Explanations  are  sufficiently  comprehensive  to  permit  use 
of  the  task  analysis  procedures. 


n 

SA 

A 

U 

0 

SO 

A  +  R 

32 

.03 

.63 

.16 

.19 

.00 

A 

16 

.06 

.63 

.19 

.13 

.00 

R 

16 

.00 

.63 

.13 

.25 

.00 

Eq 

NEq 

A 

6 

.00 

.67 

.17 

.17 

.00 

Course 

R 

A 

6 

10 

.00 

.10 

.50 

.60 

.33 

.20 

.17 

.10 

.00 

.00 

R 

10 

.00 

.70 

.10 

.20 

.00 

I  Ex 

A 

8 

.13 

.63 

.13 

.13 

.00 

Experience  Level 

Ex 

R 

A 

8 

8 

.00 

.00 

.38 

.63 

.25 

.25 

.38 

.13 

.00 

.00 

R 

8 

.00 

.88 

.00 

.13 

.00 

SR 

12 

.00 

.67 

.00 

.25 

.08 

TABLE  C-1  (CONTINUED) 


It«m  42.  Tb6  handbook  provides  an  adequate  number  and  range  of  examples. 


n 

SA 

A 

U 

0 

SO 

A  4^  R 

32 

.06 

,47 

.19 

.25 

.03 

A 

16 

.13 

.31 

.19 

.31 

.06 

R 

16 

.00 

.63 

.19 

.19 

.00 

Eq 

NEq 

A 

6 

.00 

.33 

.17 

.50 

.00 

Course 

R 

A 

6 

10 

.00 

.20 

.50 

.30 

.33 

.20 

.17 

.20 

.00 

.10 

R 

10 

,00 

.70 

.10 

.20 

.00 

lEx 

A 

8 

.25 

.25 

.25 

.25 

.00 

Experience  Level 

R 

A 

8 

8 

.00 

.00 

.38 

.38 

.38 

.13 

.25 

.38 

.00 

.13 

LX 

R 

8 

.00 

.88 

.00 

.13 

.00 

SR 

12 

.00 

.58 

.08 

.25 

.08 

Item  13.  Handbook  examples  provide  adequate  supoort  for  the  textu**! 
explanations. 


n 

SA 

A 

U 

0 

SO 

A  ♦  R 

31 

.03 

.58 

.10 

.23 

.06 

A 

16 

.06 

.63 

.13 

.06 

.06 

R 

15 

.00 

.53 

.07 

.33 

.07 

Eq 

NEq 

A 

6 

.00 

.50 

.17 

.17 

.17 

Course 

R 

A 

6 

10 

.00 

.10 

.67 

.70 

.00 

.10 

.17 

.10 

.17 

.00 

R 

9 

.00 

.44 

.11 

.44 

.00 

lEx 

Ex 

A 

8 

.13 

.63 

.13 

.00 

.13 

Experience  Level 

R 

A 

8 

8 

.00 

.00 

.50 

.63 

.00 

.13 

.38 

.25 

.13 

.00 

R 

7 

.00 

.57 

.14 

.29 

,00 

SR 

12 

.00 

.67 

.08 

.00 

.25 

TABLE  C-1  (CONaUOEO) 


Itm  32.  Tht  txan^lts  providtd  «rt  rtllstic  and  imningful. 


A 

R 


Courst 


Exptrltnca  Lavel 
SR 


n 

SA 

A 

U 

0 

32 

.09 

.69 

.09 

.13 

16 

.19 

.56 

.19 

.06 

16 

.00 

.81 

.00 

.19 

c«  * 

R 

6 

.17 

.67 

.00 

.17 

6 

.00 

.67 

.00 

.33 

NE,{ 

10 

.20 

.50 

.30 

.00 

10 

.00 

.90 

.00 

.10 

lEx* 

8 

.25 

.38 

.25 

.13 

8 

.00 

.75 

.00 

.25 

Ex  p 

8 

.13 

.75 

.13 

.00 

8 

.00 

.88 

.00 

.13 

12 

.00 

.75 

.00 

.17 

SO 
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TABLE  C-2  COMPARATIVE  EVALUATION:  PROPOSED  VS.  CURRENT  PROCEDURES 
(STEPS  CURRENTLY  PERFORMED}* 


TABLE  C-2  (COMPLETED) 


PROPOSED  PROCEDURr 


STEP 

BETTER  THAN 

SAME  AS 

WORSE  THAN 

n 

12 

identify 

SUBTASKS 

(TASK 

DIAGRAMS) 

.75 

.13 

.13 

8 

13 

SMS 

REVIEW 

.60 

.40 

.00 

5 

14 

REVISE 

SUBTASKS 

.83 

.17 

.00 

6 

15 

ANALYZE 

SUBTASKS 

.63 

.25 

.13 

8 

16 

IDENTIFY 

SUPPORTING 

SKILLS 

.45 

.45 

.10 

n 

17 

IDENTIFY 

SUPPORTING 

KNOWLEDGES 

.42 

.50 

.08 

12 

18 

SMS 

REVIEW 

.59 

.41 

.00 

7 

19 

REVISE 

SUPPORTING 

SKILLS  AND 
KNOWLEDGE 

.55 

.44 

.00 

9 

20 

DOCUMENT 

TASK 

ANALYSIS 

.62 

.25 

.13 

8 
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TABLE  D-3  ACCURACY/COMPLETENESS  OF  TASK  PERFORMANCE 
AND  TASK  knowledge  ANALYSES 


TASK  PERFORMANCE 


SA 

A 

U 

0 

SO 

TASK  ANALYSIS  ACCURATE  {n»16) 

.19 

.19 

.06 

.50 

.06 

TASK  ANALYSIS  COMPLETE  (n*16) 

.19 

.19 

.00 

.50 

.13 

X 

.19 

.19 

.03 

.50 

.10 

TASK 

KNOWLEDGE 

SA 

A 

U 

0 

SO 

TASK  .V;«li3IS  ACCURATE  {n«16) 

.19 

.31 

.00 

.38 

.13 

TASK  ANALYSIS  COMPLETE  (n«16) 

.19 

.31 

.00 

.38 

.13 

X 

.9 

.31 

.00 

.38 

.13 

TABLE  D-4  ACCURACY/COMPLETENESS  OF  EQUIPMENT  ORIENTED 
AND  NONEQUIPMENT  ORIENTED  TASK  ANALYSES 

[Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative] 

[Key:  Strongly  Agree  (SA);  Agree  (A);  Undecided  (U);  Disagree  (D):  Strongly 
Disagree  (SD)] 


EQUIPMENT  ORIENTED  COURSES 


SA 

A 

U 

D 

SD 

TASK  ANALYSIS  ACCURATE  (n=12) 

.08 

.42 

.08 

.42 

.00 

TASK  ANALYSIS  COMPLETE  {n=12) 

.08 

.33 

.00 

.50 

.08 

X 

.08 

.38 

-04 

.46 

.04 

NONEQUIPMENT  ORIENTED  COURSES 


SA 

A 

U 

D 

SD 

TASK  ANALYSIS  ACCURATE 

(n.20) 

.25 

.15 

-00 

.45 

.15 

TASK  ANALYSIS  COMPLETE 

(n=20) 

.25 

-20 

.00 

.40 

.15 

X 

.25 

.18 

.00 

.43 

.15 

table  d-5  quality  of  task  analyses  (overall) 

[Proportion  of  Respondents  Selecting  Each  Alternative] 

[Key:  Excellent  (E);  Very  Good  (VG);  Good  (G);  Fair  (F);  Poor  (P)] 

E  VG  G  F  P 
TASK  ANALYSIS  QUALITY  {n-32)  .16  .16  .13  .35  .22 


I 


TABLE  0-6  OVERALL  QUALITY  OF  TASK  PERFORMANCE  AND 
TASK  KNOWLEDGE  ANALYSES 

[Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative] 
[Key:  Excellent  {£};  Very  Goad  (V6);  Good  (G);  Fair  (F);  Poor  (p)] 


task  performance 

E 

VG 

G 

F 

P 

TASK  ANALYSIS  QUALITY  (n-16)  .13 

.19 

.06 

.50 

.13 

TASK  KNOWLEDGE 

E 

VG 

G 

F 

P 

TASK  ANALYSIS  QUALITY  (n-16)  .ig 

.13 

.19 

.19 

.31 
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TABLE  0-7  OVERALL  QUALITY  OF  EQUIPMENT  ORIENTFD 
AND  NONEQUIPMENT  ORIENTED  TASK  ANALYSES 

[Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative] 
[Key:  Excellent  (E);  Very  Good  (VG);  Good  (G);  Fair  (F);  Poor  (P)] 


EQUIPMENT  ORIENTED  COURSES 

E  VG  G 

TASK  analysis  QUALITY  (n=12)  .00  .33  .17 


F  P 

.33  .17 


NONEQUIPMENT  ORIENTED  COURSES 
E  VG 

TASK  ANALYSIS  QUALITY  {n=20)  .20  .05 


G  F  0 

.10  .35  .25 
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TABLE  0-8  UTILITY  OF  TASK  ANALYSES  DOCUMENTATION 

tCell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative] 

[Key:  Strongly  Agree  (SA);  Agree  (A);  Undecided  (U);  Disagree  (D);  Strongly 
Disagree  (SO)] 


FACILITATE  DEVELOPMENT  OF  OBJECTIVES 
AND  TEST  ITEMS  (n«32) 

FACILITATE  DEVELOPMENT- OF  POI 
(PART  I)  (n»32) 

FACILITATE  DEVELOPMENT  OF  POI 
(PART  II)  (n=32) 

FACILITATE  DEVELOPMENT  OF  OBJECTIVES 
AND  TEST  ITEMS  BY  INDEPENDENT 
SMSs  (n-32) 

INSURE  THAT  ONLY  "NEED  TO  KNOW" 
INFORMATION  IS  INCLUDED  IN 
COURSE  (n-32) 

facilitate  interpretation  of  FIELD 
EVALUATION  REPORTS  (n-32) 


$A 

A 

U 

D 

SO 

.19 

.35 

.16 

.19 

.13 

.22 

.47 

.03 

.19 

.10 

.19 

.41 

.06 

.22 

.13 

.16 

.38 

.13 

.19 

.15 

38 

.53 

.00 

.06 

.03 

,16 

.75 

.03 

.03 

.03 

22 

.48 

.07 

.15 

.10 
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TABLE  0-9  UTILITY  OF  TASK  PERFORMANCE  ANO  TASK  KNOWLEDGE  DOCUMENTATION 
[Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative] 
[Task  Performance  (TP,  Task  Knowledge  (TP)] 

[Cell  Frequency;  n»16] 

[Key:  Strongly  Agree  (SA),  Agree  (A),  Undecided  (U),  Disagree  (0), 
Strongly  Disagree  (SO)]  *  ' 


f: 

f 


SA 

A 

U 

0 

SO 

facilitate  development 

OF  OBJECTIVES  ANO  TEST 

TP 

.19 

.25 

.25 

.19 

.13 

items 

TK 

.19 

.44 

.06 

.19 

.13 

facilitate  DEVELOPMENT  OF 

POI  (PART  I) 

TP 

.25 

.50 

.00 

.19 

.06 

TK 

.19 

.44 

.06 

.19 

.13 

facilitate  DEVELOPMENT  OF 

POI  (PART  II) 

TP 

.19 

.44 

.00 

.31 

.06 

TK 

.?9 

.38 

.13 

.13 

.19 

facilitate  DEVELOPMENT  OF 
OBJECTIVES  ANO  TEST  ITEMS 

TP 

.13 

.38 

.19 

.25 

.06 

BY  INDEPENDENT  SMS 

TK 

.19 

.38 

.06 

.13 

.25 

INSURE  THAT  ONLY  "NEED  TO 
KNOW"  INFORMATION  IS 

TP 

-38 

.56 

.00 

.06 

.00 

INCLUDED  IN  COURSE 

TK 

.38 

.50 

.00 

.06 

.06 

facilitate  interpretation 

OF  FIELD  EVALUATION  REPORTS 

TP 

.19 

.69 

.06 

.06 

.00 

TK 

.13 

.81 

.00 

.00 

.06 

X 

TP 

.22 

.47 

.08 

.18 

.05 

TK 

.21 

.49 

.05 

.12 

.14 

I 
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TABLE  0-10  UTILITY  OF  DOCUMENTATION  FOR  EQUIPMENT  ORIENTED 
ANO  NONEQUIPMENT  ORIENTED  COURSES 

[Cell  Entrfes:  Proportion  of  RtsponOonts  Selecting  Each  Alternative] 

[Equlpeient  Oriented  (Eq.)«  Nonequlpment  Oriented  (NCq.)] 

[Key:  Strongly  Agree  {$A);  Agree  (A);  Undecided  (U);  Disagree  (0); 
Strongly  Disagree  (SO)] 


SA 

A 

U 

D 

SO 

FACILITATE  DEVELOPMENT  OF 
OBJECTIVES  AND  TEST 

n*?2 

Ed 

.08 

.67 

.08 

.17 

.00 

ITEMS 

n*20 

NEq 

.25 

.15 

.20 

.20 

.20 

FACILITATE  DEVELOPMENT 

OF  POl  (PART  I) 

n*12 

£q 

.17 

.75 

,00 

.08 

.00 

n*20 

NEq 

.25 

.30 

.05 

.25 

.15 

FACILITATE  DEVELOPMENT 

OF  POI  (PART  II) 

n^l2 

Eq 

.08 

.58 

.00 

.33 

.00 

n«20 

NEq 

,25 

.30 

.10 

.16 

.20 

FACILITATE  DEVELOPfKNT 

OF  OBJECTIVES  ANO  TEST 

n*72 

Eq 

.00 

.75 

.08 

.08 

.08 

ITEMS  BT  INDEPENDENT 

SMSs 

n*20 

NEq 

.25 

.15 

.15 

.25 

.20 

INSURE  THAT  ONLY  "NEED 

TO  KNOW"  INFORMATION  IS 

n«12 

Fq 

.08 

.92 

.00 

.00 

.00 

INCLUDED  IN  COURSE 

n*20 

NEq 

.55 

.30 

.00 

.10 

.05 

FACILITATE  INTERPRETATION 
OF  FIELD  EVALUATION 

n»)2 

Eq 

.08 

.63 

.08 

.00 

.00 

REPORTS 

n»20 

NEq 

.20 

.70 

.00 

.05 

.05 

7 

n*12 

er. 

.08 

.75 

,04 

.11 

n*20 

NEq. 

.29 

.32 

.08 

.17 

.14 
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TABLE  D-11  JUDGMENTS  REGARDING  HANDBOOK  VALIDITY 

[Analysts  »  A;  Reviewers  *  R;  Equipment  -  Eq;  Non-Equipment  -  NEq;  Experienced  -  Ex; 
Inexperienced  -  lEx;  Senior  Review  Teams  -  SR] 

[Key:  Strongly  Agree  (SA);  Agree  (A);  Undecided  (U);  Disagree  (o);  Strongly  Disagree  (SO' 
n  ■  number  of  entries] 

[Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative] 


Item  4.  The  task  analysis  documentation  generated  in  applying  handbook 
procedures  would  be  useful  in  developing  learning  objectives. 


Course 


Experience  Level 


n 

SA 

A 

U 

D 

SO 

32 

.28 

.59 

.06 

.06 

.00 

16 

.31 

.56 

.06 

.06 

.00 

16 

.25 

.63 

.06 

.06 

.no 

Eq 

A 

6 

.33 

.67 

.00 

.00 

.00 

R 

6 

.33 

.50 

.17 

.00 

.00 

NEq 

A 

10 

.30 

.50 

.10 

.10 

,00 

R 

10 

.20 

.70 

.00 

.10 

.00 

I  Ex 

A 

8 

.38 

.50 

.13 

.00 

.00 

R 

8 

.13 

.63 

.13 

.13 

.00 

Ex 

A 

8 

.25 

.63 

.00 

.13 

.00 

R 

8 

.38 

.63 

.00 

.00 

.00 

11 

.27 

.55 

,09 

.09 

.00 

Item  2,  Task  analysis  documentation  produced  using  handbook  procedures 

would  be  useful  in  the  preparation  of  written  tests  and  performance 
checklists. 


Course 


Experience  Level 


n 

SA 

A 

U 

D 

SO 

32 

.06 

.38 

.13 

.41 

.03 

16 

.06 

.31 

.19 

.38 

.06 

16 

.06 

.44 

.06 

.44 

.00 

Eq 

A 

6 

.00 

.33 

.17 

.33 

.17 

R 

6 

.00 

.50 

.17 

.33 

.00 

NEq 

A 

10 

.10 

.30 

.20 

.40 

.00 

R 

10 

.10 

.40 

.00 

.50 

.00 

lEx 

A 

8 

,13 

.13 

.25 

.50 

.00 

R 

8 

.00 

.38 

.13 

.50 

.00 

Ex 

A 

8 

.00 

.50 

.13 

.25 

.13 

R 

8 

.13 

.50 

.00 

.38 

.00 

12 

.17 

.42 

.17 

.25 

.00 

TABLE  0-n  (CONTINUED) 


Item  1.  The  type  of  task  analysis  docimntatlon  described  In  the  handbook 
would  be  useful  In  Interpreting  and  acting  upon  Field  Evaluation 
Reports . 


Course 


Experience  Level 


n 

SA 

A 

U 

0 

SO 

31 

,10 

.55 

.29 

.06 

.00 

15 

.07 

.60 

.27 

.07 

.00 

16 

.13 

.50 

.31 

.06 

.00 

Eq 

A 

6 

.17 

.83 

.00 

.00 

.00 

R 

6 

.17 

.50 

.33 

.00 

.00 

NEq 

A 

9 

.00 

.44 

.44 

.11 

.00 

R 

10 

.10 

.50 

.30 

.10 

.00 

lEx 

A 

8 

.00 

.50 

.50 

.00 

.00 

R 

8 

.13 

.50 

.38 

.00 

.00 

Ex 

A 

7 

.14 

.71 

.00 

.14 

.00 

R 

8 

.13 

.50 

.25 

.13 

.00 

12 

.08 

.50 

.17 

.17 

.08 

Item  6.  The  use  of  the  procedures  described  In  the  handbook  would  result 
in  complete  and  thorough  task  analyses. 


Experience  Level 


n 

SA 

A 

U 

D 

SD 

31 

.16 

.32 

.32 

.19 

.00 

16 

.25 

.19 

.31 

.25 

.00 

15 

.07 

.47 

.33 

.13 

.00 

Eq 

A 

6 

.33 

.17 

.50 

.00 

.00 

R 

6 

.00 

.50 

.33 

.17 

.00 

NEq 

A 

10 

.20 

.20 

.20 

.40 

.00 

R 

9 

.11 

.44 

.33 

.11 

.00 

I  Ex 

A 

8 

.25 

.25 

.13 

.38 

.00 

R 

a 

.00 

.50 

.25 

.25 

.00 

Ex 

A 

8 

.25 

.13 

.50 

.13 

.00 

R 

7 

.14 

.43 

.43 

.00 

.00 

12 

.00 

.58 

.17 

.17 

.08 

TABLE  0-11  (OmETEO) 


Ittm  9.  Tht  proctdures  described  In  tht  handbook.  If  properly  applied, 
would  result  in  an  accurate  task  analysis. 


n 

SA 

A 

U 

0 

SO 

A  4^  R 

31 

.16 

.52 

.23 

.10 

.00 

A 

16 

,19 

.56 

.13 

.13 

.00 

R 

15 

.13 

.47 

.33 

.07 

.00 

Eq 

A 

6 

.17 

.67 

.17 

.00 

.00 

Course 

R 

6 

.00 

.50 

.50 

.00 

.00 

NEq 

A 

10 

.20 

.50 

.10 

.20 

.00 

R 

9 

.22 

.44 

.22 

.11 

.00 

lEx 

Ex 

A 

8 

.25 

.50 

.25 

.00 

.00 

Experience  Level 

R 

A 

8 

8 

.00 

.13 

.63 

.63 

.25 

.13 

.13 

.13 

.00 

.00 

R 

7 

.29 

.29 

.43 

.00 

.00 

SR 

12 

.00 

.75 

.00 

.17 

.08 
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TABLE  D-12  NECESSITY  OF  STEP 
(Proportion  Responding  Positively)* 


1  breakup 

GLOBAL 

STS/CTS 

.89 

2  LIST  ALL  TASK 

PERFORMANCES  AND 

task  knowledges 

.89 

3  CONVERT  TO 

behavioral 

STATEMENTS 

.85 

A  SMS 

review 

.75 

5  revise 
behavioral 

STATEMENTS 

.74 

6  CONVERT  TO 

preliminary 

CRITERION 

OBJECTIVES 

(PCO) 

.82 

2  SMS 

review 

.85 

8  REVISE 

PCO 

.82 

8  DOCUMENT 

PCO 

.85 

ID  DETERMINE 
type  of  TASK 

reflected  in 

PCD 

.82 

11  SELECT 

analysis 

technique 

.82 

12  identify 
subtasks 
(task  DIA¬ 
GRAMS) 

.35 

13  SMS 

REVIEW 

.78 

•INNOVATION  EVALUATION  SURVEY:  SCALE  1 
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TABLE  0-12  (CONCLUDED) 


STEP 

14  REVISE 
SUBTASKS 

15  ANALYZE 
SUBTASKS 

16  IDENTIFY 
SUPPORTING 
SKILLS 

17  IDENTIFY 
SUPPORTING 
KNOWLEDGES 

18  SMS 
REVIEW 

19  REVISE 
SUPPORTING 
SKILLS  AND 
KNOWLEDGES 


.74 

.82 

.93 

.89 

.82 

.85 


20  DOCUMENT 

TASK  .85 

ANALYSIS 
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TABLE 

TASK  PERFORMANCE 
TASK  KNOWLEDGE  ( 

WEIGHTED  X 

EQUIPMENT  (n*6) 
NONEQUIPMENT  (n- 


E-1  JUDGED  CORRESPONDENCE  OF  TASK  ANALYSES  (STRINGENT) 


PCOs 

SUBTASKS 

SKILLS/KNOWLEDGES 

OVERALL 

(n-8) 

.75 

.25 

.25 

.25 

n«8) 

.50 

.38 

.25 

.38 

.63 

.32 

.25 

,31 

.67 

.16 

.00 

.16 

10) 

.60 

.40 

.40 

.40 

WEIGHTED  X 


.63 


.32 


.25 


.31 


TABLE  £-2 


task  performance  (n-8) 

task  knowledge  (n-B) 

WEIGHTED  T 

equipment  (n.6) 
NONEQUIPMENT  (n.io) 

WEIGHTED  J 


JUDGED  CORRESPONDENCE  OF  TASK  ANALYSIS  (LENIENT*) 


PCOs 

.88 

SUBTASKS 

.25 

SKILLS/KNOWLEDGES 

.25 

.62 

.50 

.25 

.75 

.38 

.25 

.83 

.?6 

.00 

.70 

.50 

.40 

.75 

.38 

.25 

‘LENIENT:  FUNCTIONAL  EQUIVALENCE 
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TABLE  E-3  JUDGMENTS  REGARDING  HANDBOOK  RELIABILITY 


[Analysts  «  A;  Reviewers  »  R;  Equipment  -  £q;  Non-Equipment  -  NEq;  Experienced  -  Ex; 
Inexperienced  -  lEx;  Senior  Review  Teams  -  SR] 

^  'ey:  Strongly  Agree  (SA);  Agree  (A);  Undecided  (U);  Disagree  (D);  Strongly  Disagree  (SO); 
n  -  number  of  entries] 

[Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative] 


Item  12.  Two  task  analysts,  equal  in  subject  matter  expertise  and 
field  experience,  using  handbook  procedures  and  workina 
independently,  would  produce  essentially  the  same  results. 


n 

SA 

A 

U 

0 

SO 

A  +  R 

32 

,09 

.23 

.28 

.22 

.13 

A 

16 

.19 

.19 

.19 

.25 

.19 

R 

16 

.00 

.38 

.38 

.19 

.06 

Eq 

A 

6 

.33 

.33 

.00 

.17 

.17 

Course 

R 

6 

.00 

.50 

.50 

.00 

.00 

NEq 

A 

10 

.10 

.10 

.30 

.30 

.20 

R 

10 

.00 

.30 

.30 

.30 

.10 

lEx 

A 

8 

.13 

.25 

.13 

.13 

.38 

Experience  Level 

R 

8 

.00 

.50 

.38 

.13 

.00 

Ex 

A 

8 

.25 

.13 

.25 

.38 

,00 

R 

8 

,00 

.25 

.38 

.25 

.13 

SR 

12 

.00 

.50 

.08 

.33 

.08 

Item  10.  Standardized  application  of  handbook  procedures  across  all  types 
of  courses  would  produce  high  quality  results  consistently. 


n 

SA 

A 

U 

0 

SO 

A  +  R 

31 

.03 

.35 

.32 

.19 

.10 

A 

16 

.00 

.25 

.44 

.19 

.13 

R 

15 

.07 

.47 

.20 

.20 

.07 

C  A 

6 

.00 

.33 

.33 

.17 

.17 

R 

6 

.00 

.50 

.17 

.17 

.17 

Course 

10 

.00 

.20 

.50 

.20 

.10 

NEq  „ 

9 

.11 

.44 

.22 

.22 

.00 

TCw  A 

8 

.00 

.13 

.50 

.25 

.13 

lEx  ^ 

8 

.00 

.63 

.00 

.25 

.13 

Experience  Level 

C  A 

8 

.00 

.38 

.38 

.13 

.13 

Ex  ^ 

7 

.14 

.29 

.43 

.14 

.00 

SR 

12 

.08 

.33 

.25 

.17 

,17 
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1.03 


TABLE  F-2  RELATIVE  AMOUNT  OF  TIME  CURRENTLY  SPENT  ON  EACH  STEP* 
(Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative) 

(n-14) 


STEP 

NONE 

SMALL 

MODERATE 

LARGE 

1 

BREAKUP 

aOBAL 

STS/CTS 

ITEMS 

.36 

.28 

.28 

.07 

2 

LIST  ALU 

TASK 

PERFORMANCES 

AND  TASK 
KNOWLEDGES 

.14 

.21 

.64 

.00 

3 

CONVERT  TO 
BEHAVIORAL 
STATEMENTS 

.07 

.14 

.57 

.21 

4 

SMS 

REVIEW 

.23 

.31 

.23 

.23  (n*13 

5 

REVISE 

behavioral 

STATEMENTS 

.28 

.21 

.50 

.00 

6 

CONVERT  TO 

PRELIMINARV 

CRITERION 

OBJECTIVES 

(PCO) 

.21 

.14 

.36 

.28 

7 

SMS 

REVIEW 

.25 

.17 

.50 

.08  (n-12) 

8 

REVISE 

PCO 

.21 

.28 

.36 

.14 

9 

DOCUMENT 

PCO 

.21 

.14 

.50 

.14 

10 

DETERMINE 

TYPE  OF  TASK 
REFLECTED 

IN  PCO 

.23 

.31 

.46 

.00  (n-13) 

n 

SELECT 

ANALYSIS 

TECHNIQUE 

.43 

.36 

.21 

.00 

INNOVATION  EVALUATION  SURVEY:  SCALE  2 


TABLE  F-2  (CONCLUDED) 


STEP 

NONE 

SMALL 

MODERATE 

72 

IDENTIFY 

SUPPORTING 

(TASK 

DIAGRAMS) 

.43 

.23 

.14 

13 

SMS 

REVIEW 

.64 

.14 

,21 

14 

REVISE 

SUBTASKS 

.57 

.36 

.07 

15 

ANALYZE 

SUBTASKS 

.43 

.21 

.28 

16 

IDENTIFY 

SUPPORTING 

SKILLS 

.21 

.21 

.50 

17 

IDENTIFY 

SUPPORTING 

KNOWLEDGES 

.14 

.21 

.57 

18 

SMS 

REVIEW 

.50 

.14 

.28 

19 

REVISE 
SUPPORTING 
SKILLS  AND 
KNOWLEDGE 

.36 

.28 

.36 

20 

DOCUMENT 

TASK 

ANALYSIS 

.36 

.07 

57 
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TABLE  F-3  TIME  EXPENDITURE  JUDGMENTS;  PROPOSED  VS. 

CURRENT  PROCEDURES  (STEPS  CURRENTLY  PERFORMED)* 

(Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative) 


STEP 

MORE 

1 

BREAKUP 

GLOBAL 

STS/CTS 

ITEMS 

.22 

2 

LIST  ALL  TASK 
PERFORMANCES 

AND  TASK 

KNOWLEDGE 

.33 

3 

CONVERT  TO 

BEHAVIORAL 

STATEMENTS 

.15 

4 

SMS 

.36 

REVIEW 

5 

REVISE 

BEHAVIORAL 

STATEMENTS 

.20 

6 

CONVERT  TO 

preliminary 

CRITERION 

OBJECTIVES 

(PCO) 

.19 

7 

SMS 

.11 

REVIEW 

8 

REVISE 

.10 

PCO 

9 

DOCUMENT 

.27 

PCO 

10 

determine 

TYPE  OF  TASK 
REFLECTED 

IN  PCO 

.30 

n 

SELECT 

ANALYSIS 

TECHNIQUE 

.13 

PROPOSED  PROCEDURE 


EQUAL 

LESS 

n 

.55 

.22 

9 

.42 

.25 

12 

.54 

.31 

13 

.28 

.36 

11 

.40 

.40 

10 

.36 

.45 

n 

.33 

,55 

9 

45 

.45 

n 

27 

.45 

11 

40 

.30 

10 

63 

.25 

8 

•INNOVATION  EVALUATION  SURVEY:  SCALE  3 
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TABLE  F-3  (CONCLUDED) 


12  IDENTIFY 

SUBTASKS 
(TASK  DIA¬ 
GRAMS) 

.50 

PROPOSED  PROCEDURE 

•13  .37 

8 

13  SMS 

REVIEW 

.00 

.40 

.60 

5 

14  REVISE 

SUBTASKS 

.00 

.50 

.50 

6 

15  ANALYZE 

SUBTASKS 

.37 

.25 

.37 

8 

16  IDENTIFY 
SUPPORTING 

SKILLS 

.27 

.55 

.18 

11 

17  IDENTIFY 
SUPPORTING 
KNOWLEDGES 

.25 

.58 

.17 

12 

18  SMS 

REVIEW 

.00 

.43 

.57 

7 

19  REVISE 

SUPPORTING 

SKILLS  AND 
KNOWLEDGE 

.00 

.55 

.44 

9 

20  DOCUMENT 

TASK 

ANALYSIS 

.13 

.37 

.50 

8 

•INNOVATION  EVALUATION  SURVEY:  SCALE  3 
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TABLE  F-4  JUDGMENTS  REGARDING  TIME-EFFICIENCY/COST-EFFECTIVENtSS 

(Analysts  «  A;  Reviewers  •  R;  Equipment  ■  Eq;  Non-Equipment  •  NEq;  Experienced 
Inexperienced  *  lEx;  Senior  Review  Teams  «  SR). 

(Key:  Strongly  Agree  (SA);  Agree  (A);  Undecided  (U);  Disagree  (0);  Strongly 
Disagree  (SD);  n  »  number  of  entries), 

(Cell  Entries:  Proportion  of  Respondents  Selecting  Each  Alternative). 

1 .  User/Ski 1 1  Requi rements 

Item  3.  To  be  useful •  the  handbook  would  not  require  extensive  training 
In  ISD. 


n 

SA 

A 

U 

0 

SO 

A  >  R 

32 

.13 

.53 

.06 

.22 

.06 

A 

16 

.06 

.56 

.06 

.25 

.06 

R 

16 

.19 

.50 

.06 

.19 

.06 

Eq 

A 

6 

.00 

.67 

.00 

.17 

,17 

Course 

R 

6 

.17 

.50 

.00 

.33 

.00 

NEq 

A 

10 

.10 

.50 

.10 

.30 

.00 

R 

10 

.20 

.50 

.10 

.10 

,10 

I  Ex 

A 

8 

.13 

.38 

.13 

.25 

.13 

Experience  Level 

R 

8 

,00 

.63 

.00 

.38 

.00 

Ex 

A 

8 

.00 

.75 

.00 

.25 

.00 

R 

8 

.38 

.38 

.13 

.00 

.13 

SR 

11 

.09 

.27 

.09 

.18 

.33 

Item  8.  The  handbook  would  be  most  useful  In  the  hands  of  a  skilled 
Instructor  with  extensive  experience  In  his  specialty. 


n 

SA 

A 

U 

0 

SO 

A  ♦  R 

31 

.19 

.48 

.16 

.16 

.00 

A 

16 

.25 

.44 

.19 

.13 

.00 

R 

15 

.13 

.53 

.13 

.20 

.00 

Eq 

A  6 

.33 

.50 

.00 

.17 

.00 

Course 

R  6 

.17 

.50 

.17 

.17 

.00 

NEq 

A  10 

.20 

.40 

.30 

.10 

.00 

R  9 

.11 

.50 

.11 

.22 

.00 

lEx 

A  8 

.25 

.50 

.13 

.13 

.00 

Experience  Level 

R  7 

.14 

.57 

.00 

.29 

.00 

EX 

A  8 

.25 

.38 

.25 

.13 

.00 

R  8 

.13 

.50 

.25 

.13 

.00 

SR 

12 

.42 

.25 

.08 

.17 

.08 

TABLE  F.4  (CONTINUED) 


Item  29.  Adoption  of  this  task  analysis  approach  in  this  Branch 
Mould  not  require  additional  man-power. 


N 

SA 

A 

U 

0 

SO 

A  +  R 

31 

.03 

.26 

.29 

.35 

.06 

A 

15 

.00 

.27 

.47 

.20 

.07 

R 

16 

.06 

.25 

.13 

.50 

.06 

Eq 

A 

6 

.00 

.33 

.50 

.00 

.17 

Course 

R 

6 

.00 

.17 

.17 

.67 

.00 

NEq 

A 

9 

.00 

.22 

.44 

.33 

.00 

R 

10 

.10 

.30 

.10 

.40 

.10 

lEx 

A 

7 

.00 

.29 

.57 

.14 

.00 

Experience  Level 

R 

8 

.00 

.38 

.13 

.38 

.13 

Ex 

A 

8 

.00 

.25 

.38 

.25 

.13 

R 

8 

.13 

.13 

.13 

.63 

.00 

SR 

11 

.00 

.18 

.27 

.36 

.18 

Item  26. 

These  task  analysis  procedures 

could 

be  implemented 

in 

this  Branch  with  minimal 

additional 

resourses 

N 

SA 

A 

U 

0 

SO 

A  +  R 

32 

.06 

.31 

.41 

.22 

.00 

A 

16 

.00 

.44 

.31 

.25 

.00 

R 

16 

.13 

.19 

.50 

.19 

.00 

Eq 

A  6 

.00 

.50 

.50 

.00 

.00 

Course 

R  6 

.00 

.17 

.67 

.17 

.00 

NEq 

A  10 

.00 

.40 

.20 

.40 

.00 

R  10 

.20 

.20 

.40 

.20 

,00 

Ex 

A  8 

.00 

.50 

.25 

.25 

.00 

Experience  Level 

R  8 

.25 

.00 

.50 

.25 

.00 

I  Ex 

A  8 

.00 

.38 

.38 

.25 

.00 

R  8 

.00 

.38 

.50 

.13 

.00 

SR 

12 

.00 

.17 

.17 

.50 

.17 
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TABLE  F-4  (CONTINUED) 


3.  Implementation  Issues 


Item  20.  It  woulO  maKe  good  sense  to  do  this  type  of  task  analysis 
when  a  course  Is  undergoing  major  revision. 


n 

SA 

A 

U 

0 

SO 

A  ♦  R 

32 

.22 

.63 

.06 

.06 

.03 

A 

16 

.25 

.56 

.13 

.00 

.06 

R 

16 

.19 

.69 

.00 

.13 

.00 

Eq 

A 

6 

.33 

.67 

.00 

.00 

.00 

Course 

R 

6 

.17 

.50 

.00 

.33 

.00 

NEq 

A 

10 

.20 

.50 

.20 

.00 

,10 

R 

10 

.20 

.80 

.00 

.00 

,00 

I  Ex 

A 

8 

.38 

.50 

.13 

.00 

.00 

Experience  Level 

R 

8 

.13 

.75 

.00 

.13 

.00 

Ex 

A 

8 

.13 

.63 

.13 

.00 

.13 

R 

8 

.25 

.63 

.00 

.13 

,00 

SR 

12 

.25 

.42 

.08 

.17 

.08 

Item  23.  The  handbook  task  analysis  procedures  described  would 
be  best  applied  in  developing  new  courses. 


A  +  R 

n 

SA 

A 

U 

0 

SO 

A 

R 

32 

.16 

.50 

.13 

.19 

.03 

16 

.25 

.44 

.13 

.13 

.06 

16 

.06 

.56 

.13 

.25 

.00 

£q 

A 

6 

.33 

.50 

.17 

.00 

.00 

Course 

R 

6 

.17 

,17 

.17 

.50 

.00 

NEq 

A 

10 

.20 

.40 

.10 

.20 

.10 

R 

10 

.00 

.30 

.10 

.10 

.00 

lEx 

A 

8 

.25 

.50 

.13 

.00 

.13 

Experience  Level 

R 

8 

.00 

.50 

.13 

.38 

.00 

Ex 

A 

8 

.25 

.38 

.13 

.25 

.00 

R 

8 

.13 

.63 

.13 

.13 

.00 

SR 

12 

.08 

.50 

.08 

.25 

.08 
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TABLE  F-4  (CONTINUED) 


Item  7.  The  procedures  described  would  be  applicable  to  the  Tull 
range  of  courses  taught  by  ATC. 


n 

SA 

A 

U 

0 

SO 

A  +  R 

31 

.03 

.39 

.26 

.29 

.03 

A 

16 

.06 

.31 

.31 

.25 

.06 

R 

15 

.00 

.47 

.20 

.33 

.00 

Eq 

A 

6 

.17 

.50 

.17 

.17 

.00 

Course 

R 

6 

.00 

.67 

.17 

.17 

.00 

NEq 

A 

10 

.00 

.20 

.40 

.30 

.10 

R 

9 

.00 

.33 

.22 

.44 

.00 

I  Ex 

A 

8 

.00 

.25 

.50 

.13 

.13 

Experience  Level 

R 

8 

.00 

.38 

.13 

.50 

.00 

EX 

A 

8 

.13 

.38 

.13 

,38 

.00 

R 

7 

.00 

.57 

.29 

,14 

.00 

SR 

12 

.00 

.50 

.25 

.08 

.17 

Item  11. 

The 

handbook  would 

be  useful 

as 

part  of 

an  ISO 

OJT 

program. 

n 

SA 

A 

U 

0 

SO 

A  +  R 

32 

.41 

.41 

09 

.09 

.00 

A 

16 

.38 

.44 

06 

.13 

.00 

R 

16 

.44 

.38 

13 

.06 

.00 

Eq 

A 

6 

.17 

.67 

on 

.17 

.00 

R 

6 

.50 

.33 

17 

.00 

.00 

NEq 

A 

10 

.50 

.30 

10 

.10 

.00 

R 

10 

.40 

.40 

10 

.10 

.00 

I  Ex 

A 

8 

.38 

.38 

13 

.13 

.00 

R 

8 

.25 

.50 

13 

.13 

.00 

Ex 

A 

8 

.38 

.50 

00 

.13 

.00 

R 

8 

.57 

.29 

13 

.00 

.00 

SR 

12 

.17 

.58 

00 

.17 

.08 
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TABLE  F-4  (CONCLUDED) 


2.  Manpower  Issues 


Item  40.  Periodic  reviews  by  fellow  Subject-Matter  Specialists 
during  the  course  of  the  analysis  are  a  good  way  of 
insuring  accuracy  and  completeness. 


n 

SA 

A 

U 

D 

so 

A  +  R 

32 

.34 

.56 

.06 

.03 

.00 

A 

16 

.38 

.50 

.06 

.06 

.00 

R 

16 

.31 

.63 

.06 

.00 

.00 

Eq 

A 

6 

.33 

.67 

.00 

,00 

.00 

Course 

R 

6 

.17 

.83 

.00 

.00 

.00 

HEq 

A 

10 

,40 

.40 

.10 

.10 

.00 

R 

10 

.40 

,50 

.10 

.00 

,00 

lEx 

A 

8 

.25 

.63 

.13 

.00 

.00 

Experience  Level 

R 

8 

.13 

.75 

.13 

.00 

.00 

Ex 

A 

8 

.50 

.38 

.00 

.13 

.00 

R 

8 

.50 

.50 

.00 

.00 

.00 

SR 

11 

.36 

.63 

.00 

.00 

.00 

Item  37. 

Working  in 

two- 

■person 

teams  is 

time 

efficient 

and 

cost 

effective. 

n 

SA 

A 

U 

0 

SO 

A  ♦  R 

31 

.23 

,45 

.10 

.16 

.06 

A 

15 

.27 

.47 

.07 

.07 

.13 

R 

16 

.19 

.44 

.13 

.25 

,00 

Eq 

A 

6 

.33 

.50 

.00 

.00 

.17 

R 

6 

.00 

.33 

.00 

.17 

.00 

NEq 

A 

9 

.22 

.44 

.11 

.11 

.11 

Course 

R 

10 

.30 

.20 

.20 

.30 

.00 

lEx 

A 

7 

.43 

.14 

.14 

.14 

.14 

Experience  Level 

R 

3 

.00 

.50 

.13 

.38 

.00 

Ex 

A 

8 

.13 

.75 

.00 

.00 

.13 

SR 

R 

8 

.38 

.38 

.13 

.13 

.00 

n 

.27 

.45 

.18 

.09 

.00 
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APPENDIX  G 


PRELIMINARY  DESIGN  NOTES:  TASK  ANALYSIS  DATA  BANK 
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Overview 


This  planning  document  describes  an  automated  information  storage 
and  retrieval  system  (ISRS)  for  Air  Training  Command  (ATC)  task  analysis, 
data.  The  primary  purposes  of  the  proposed  Task  Analysis  Data  Bank 
(TADB)  are  to  eliminate  duplication  of  effort  and  to  enhance  management 
control  of  the  training  development  process.  In  the  sections  that 
follow,  the  benefits  of  providing  a  TADB  are  delineated,  the  design 
process  is  discussed,  a  system  description  is  provided,  and  an  estimate 
of  implementation  cost  is  offered.  It  should  be  noted  that  the  proba¬ 
bility  of  achieving  economy  can  be  maximized  by  considering  for  inclu¬ 
sion  in  the  TADB  all  Air  Force  (AF)  specialities  that  are  unclassifieo, 
have  more  than  300  job  incumbents,  and  consist  of  40  or  more  distinct 
tasks. 

Benefits 


ATC  Technical  Training  Centers 

0  Eliminate  redundant  analyses,  needless  duplication  of  task 
analysis  efforts,  and  enhance  management  and  evaluation  of  task 
analysis  efforts  by  requiring  the  production  and  preservation  of 
standardized  end- item  documentation. 

0  Facilitate  the  design  and  revision  of  instructional  materials 
and  tests  by  providing  an  information  base  to  support  rational, 
objective  decision-making. 

0  Improve  the  efficiency  of  course  design  and  revision  activities 
by  providing  novice  analysts  with  a  full  range  of  task  analy¬ 
sis  examples. 

ATC  Headquarters 

0  Enhance  management  and  evaluation  of  task  analysis  efforts. 

0  Enhance  management  and  evaluation  of  course  development  and 
revision  activities. 

0  Facilitate  evaluation  of  training  resources  requests. 

Other  Organizations 

0  Occupational  Measurement  Center  -  Facilitate  the  development 
and  revision  of  task  inventories  and  Speciality  Knowledge 
Tests. 

0  Extension  Course  Institute  -  Enhance  management  and  evaluation 
oi  Career  Development  Course  preparation  and  revision  efforts. 


0  Military  Personnel  Center  -  Support  preparation  of  v/ork  center 
catalogs  to  facilitate  on-the-job  training  in  critical  tasks. 

0  Management  Engineering  Agency  -  Support  development  and  evalua¬ 
tion  of  manpower  standards. 

Design  Process 


Objective  -  Design  an  automated  ISRS  for  task  analysis  data. 

Primary  Constraints  -  The  goal  was  to  develop  a  TADB  design  that 
could  be  demonstrated  and  ultimately  implemented  at  reasonable  cost. 
To  minimize  hardware  costs,  the  design  was  expected  to  propose  utili¬ 
zation  of  a  computer  main  frame  already  in  the  AF  inventory.  To  mini¬ 
mize  software  development  costs,  attention  was  to  be  focused  on  opera¬ 
tionally  proven  ISRSs. 

Candidates  -  Four  generic  hardware  systems  were  selected  for 
consideration  -  the  UNIVAC  1108,  the  Honeywell  1600,  the  Burroughs 
3500  and  the  IBM  370/155.  Four  preliminary  screening  criteria  were 
established  to  identify  individual  automated  ISRSs  for  further  consider¬ 
ation  within  the  context  of  the  TADB  design  effort.  It  was  determined 
that  the  ISRS  must 

(1)  Be  compatible  with  one  of  the  four  main  frames  previously 
mentioned. 

(2)  Accommodate  on-line  entry,  editing,  and  perusal  of  task 
analysis  data  by  non-programmers. 

(3)  Provide  a  full  range  of  report  generation  capabilities. 

(4)  Have  a  terminal  and  data  base  security  protection  system. 

Of  the  46  ISRSs  surveyed  by  Fife,  Rankin,  Fong,  ^»alker,  and  Marron 
(1974),  five  were  UNlVAC-compatible  (see  Table  G-1  ). 

Final  Trades  and  Selected  Approach  -  Of  the  four  hardware  systems 
considered,  a  UNIVAC  1108,  such  as  the  one  located  at  Brooks  Air  Force 
Base,  is  preferred  for  the  proposed  application.  First  of  all,  this 
system  accommodates  research  and  developiuent  (R  and  D)  activities  quite 
nicely.  Secondly,  several  operationally  proven  ISRS  are  compatible  with 
this  main  frame.  Thirdly,  the  fact  that  the  Coiupretiensive  Occupational 
Data  Analysis  Programs  (CODAP)  system  is  already  resident  on  a  UNIVAC 
system  would  facilitate  interfacing  the  CODAP  and  task  analysis  data 
bases  in  the  event  that  subsequent  analyses  indicate  ttie  utility  of  such 
an  interface.  Lastly,  the  liardware  procured  and  the  software  capabil¬ 
ities  developed  to  support  demonstration  and  evaluation  of  the  TADB 
concept  could  be  utilized  in  other  R  and  D  applications. 
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TABLE  G-1 


UNIVAC 
System  Name 
DML 

EM I SARI 

GIM 

MIRADS 

System  2000 


COMPATIBLE  ISRSs 


Originator 

Computer  Sciences  Corp.  (CSC) 
Information  NetvJork  Div. 

650  N.  Sepulveda  Blvd. 

El  Segundo,  CA  90245 

Mathematics  and  Computation 
Laboratory 

Office  of  Emergency  Prepared¬ 
ness 

7706  Old  Springhouse  Rd. 
McLean,  VA  22101 

TRW  Systems  Group 
7600  Col  shire  Drive 
McLean,  VA  22101 

Data  Center  Division 
NASA 

Marshall  Space  Flight  Center, 
AL  35812 

MR  I  Systaiis  Corp. 

P.O.  Box  9968 
Austin,  TX  78766 
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Of  the  five  operationally  proven  ISRS  conipatible  with  the  UNIVAC 
1108,  System  2000  is  the  preferred  ISRS  for  the  proposed  application  for 
two  reasons.  First,  it  provides  a  great  number  of  implementation  alter¬ 
natives  in  the  event  that  the  TADB  demonstration  proves  successful. 
That  is.  System  2000  is  compatible  with  a  large  number  of  main  frames. 
Second,  it  is  immediately  available  at  no  cost  on  the  preferred  computer 
system. 

To  summarize  briefly,  the  selected  approach  calls  for  utilizing 
an  AF  owned  UNIVAC  1108  and  modifying  Systeffi  20UU  software  to  support 
deinonstration  and  evaluation  of  the  TADB. 

System  Description 

Hardware  -  The  system  configuration  will  consist  of  a  UNIVAC  1108 
central  main  frame  and  33  UNIVAC  UNISCOPE  200  timesharing  terminals  -  28 
with  hard-copy  printers.  Tlie  terminals  will  be  distributed  across  sites 
as  shown  in  Figure  G-1.  At  Lackland,  Lowry,  Sheppard,  Keesler,  and 
Chanute,  preliminary  plans  call  for  providing  a  printer  terminal  for 
each  Technical  Training  Group  and  a  terminal  without  print  capability 
for  the  cognizant  headquarters  activity. 

Software 

Overview  -  System  2000  is  a  hierarchical  data  base  management 
system.  The  user  specifies  the  data  fields,  structure,  and  data  inter- 
rel  ationshi  ps.  It  is  an  English-like,  user-oriented  access  language 
with  a  flexible  report  writing  system,  and  it  is  easily  adaptable  to 
changing  requirements.  Indexes  are  used  for  rapid,  efficient  selec¬ 
tion  of  qualified  data.  System  2000  is  a  comprehensive  data  base 
management  system  and  is  useful  in  batch  and  on-line  applications  to 
fulfill  the  data  processing  requirements  of  users.  It  provides  for 
data  base  design  and  definition;  data  base  creation;  data  manipulation; 
and  data  base  administration. 


System  2000  allows  the  data  base  administrator  and  uesigntr  to 
carry  a  users  requirement  through  to  full  implementation  in  a  minimal 
amount  of  time.  The  basic  components  of  the  data  base  definition  are 
data  items  (called  data  elements)  and  data  records  (called  repeating 
groups).  The  maximum  sizes  are:  16M  repeating  groups,  250  character 
fields,  32  levels  and  500  fields.  The  data  base  administrator  hiay 
specify  items  to  be  key.  Data  represented  by  key  items  are  indexed 
to  enhance  the  data  qualifications  and  access  process.  Overflow  areas 
are  available  for  name  and  text  fields  to  allow  space  savings. 

Initial  and  incremental  data  base  loading  is  accomplished  with 
the  Natural  Language  or  Procedural  Language  Interface  (PLI).  Data 
loading  may  be  accomplished  through  the  use  of  Natural  Language  commands 
or  the  user  may  employ  tiie  LOAD  command  to  load  data  from  a  prede¬ 
fined  data  file.  Data  base  loading,  editing  and  verification  may 
be  accomplished  through  the  use  of  tlie  PLI. 
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Figure  G-1  TADB  Network 


The  PLI  enables  the  user  to  manipulate  data  In  a  System  2000  data 
base  from  a  Procedural  Language  such  as  COBOL,  FORTRAN,  PL/1,  and 
Assembly  Language.  The  PLI  is  available  for  complex  data  retrieval, 
editing,  verification,  and  loading  of  data  and  when  multiple  uata  base 
access  is  required.  A  maximum  of  four  multiple  simultaneous  data  bases 
can  be  created  using  the  PLI. 

Natural  Language  provides  a  complete  set  of  canmands  for  retrieving 
and  updating  data  that  require  no  special  programming  skills.  The 
Natural  Language  Module  includes  Imniediate  Access  Syntax  ana  Queue 
Access  Syntax.  Imniediate  access  indicates  that  oata  modifications  are 
performed  immediately.  Queue  access  stacks  tlie  coniinands  and  performs 
the  operations  only  after  the  issuance  of  a  TERMINATE  con.mand.  The 
advantage  of  the  queue  method  is  the  resulting  savings  accrued  v.'itit  many 
data  modifications.  Multiple  simultaneous  access  is  available  for 
retrieval  but  not  for  updates.  Frequently  used  commands  may  be  defined 
as  strings  and  will  be  executed  by  typing  tiie  string  designator. 

The  Report  Writer  feature  allows  the  end  user  to  prepare  report 
specifications  following  a  quickly  learned  report  fonnat.  This  feature 
is  used  to  perforni  totaling,  suttotaling  and  mathematical  calculations. 
The  results  can  be  supplied  in  user  designated  format.  The  reports  may 
be  produced  in  tii.iesharing  or  held  as  output  for  tlie  printer. 

System  2000  provides  a  wide  range  of  features  to  insure  data  base 
security.  A  complete  activity  audit  is  maintained.  Security  is  pro¬ 
vided  at  remote  terminals  through  terminal  identification  and  passwords. 
Password  security  is  applied  at  the  system  data  base  record  and  itei„ 
levels.  Password  holders  can  be  restricted  on  an  i tem-by-i tern  basis, 
as  to  retrieval  authority,  update  authority,  data  selections  or  any 
combination.  Validations  are  checked  during  update  processing.  Tlie 
Update  Log  file  records  all  update  transactions  against  a  data  base 
and  may  be  used  to  backtrack  from  ttie  current  data  base  to  an  earlier 
version.  Data  base  backup  is  easily  accomplished  through  magnetic 
tape  copies  or  tlie  use  of  an  on-line  file  which  holds  the  latest  up¬ 
dated  version.  Data  base  restructuring,  including  additions  or  dele¬ 
tions  of  indexes,  changing  data  or  record  oescri pt ions,  or  modifying 
data  structures  is  also  available. 

Data  Base  -  The  TADB  is  designed  as  a  real-time,  on-line,  inter¬ 
active  ISRS.  It  will  support  multiple  con.puter  terminals  ana  will 
provide  quick  on-line,  interactive  access  to  all  information  in  the  data 
base  for  authorized  users.  Tiie  proposed  data  base  will  provide  a  cost 
effective  system  which  will  eliminate  duplication  of  training  develop¬ 
ment  efforts  and  enhance  the  control  of  training  programs. 

Ttie  TADB  is  designed  to  contain  the  following  oata  elements: 
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COURSE  NUMBER  -  a  unique  identifier  assigned  to  each  course  of  instruc¬ 
tion 

STS/CTS  NUMBER  -  training  standard  identifier 
STS/CTS  REFERENCE  -  of  an  individual  task 
DUTY  -  to  be  addressed  by  the  task 
DATE  -  task  schedule  preparation  date 
ANALYST  -  developer  of  the  training  session 

PROFICIENCY  CODE  -  level  of  proficiency  required  to  coniplete  the  task 

STS  TASK  STATEMENT  -  subject  matter  addressed  by  the  task 

BEHAVIOR  -  the  task  to  be  perforaed 

CONDITIONS  -  under  which  the  task  is  performed 

STANDARDS  -  to  be  met  by  task  performance 

REFERENCES  -  list  of  materials  which  may  be  useful  for  learning  the 
task 

STEP  NUMBER  -  sequential  number  identifying  major  areas  of  the  task 
SUBTASK/DECISION  -  component  procedure  of  the  task 

DECISION  YES  -  Step  reference  designator  for  "yes"  decision  question 
DECISION  NO  -  step  reference  designator  for  "no"  decision  question 
GO  TO  STEP  -  step  reference  designator  for  sequential  subtask  progres¬ 
sion 

SKILLS  -  required  to  perform  the  subtask 
KNOWLEDGES  -  required  to  perform  the  subtask 

The  proposed  data  base  consists  of  814  characters  per  unique 
entry  (see  Table  G-2  ).  These  data  elefnents  contain  all  the  information 
necessary  to  provide  effective  and  efficient  training  development 
and  control  procedures.  Data  represented  by  key  items  are  indexed 
to  establish  access  criteria  and  to  establisti  the  data  qualifications 
and  access  process.  Through  the  declaration  of  keyed  elements  tiie 
user  has  more  efficient  access  to  items  in  the  data  base.  Cross  refer¬ 
encing  of  data  is  achieved  through  key  item  indexes  which  provide 
flexible  data  searciiing  and  browsing  capabilities.  Keyed  elements 
in  the  TADB  insure  the  effectiveness  of  retrieval  and  update  activi¬ 
ties  by  establishing  commonalities  of  course  nun.ber,  STS  number,  STS 
reference,  duty,  proficiency,  task  statement,  behavior,  conditions, 
subtask,  skills,  and  knowledges.  Most  importantly,  training  developi.ient 
personnel  and  researchers  will  have  the  capability  to  examine  task, 
subtask,  and  skill  and  knowledge  commonalities  across  courses  and  within 
and  across  career  fields.  The  structure  of  TADB  records  is  shown  in 
Figure  G-2  and  a  sample  record  is  shown  in  Figure  G-3.  An  estimated  2UU 
courses  v/ill  comprise  the  data  base. 

Performance  Requirement  -  The  TADB  shall  be  operational  froi;.  0600 
to  2400  hours  on  Monday  through  Friday. 

Backup  Capability  -  Users  will  be  required  to  save  a  particular 
cycle  of  the  data  base  and  maintain  a  record  of  modifications  n;ade  to 
the  base  cycle  so  that  the  current  data  base  or  any  i nten.ieoi ate  version 
can  always  be  reproduced  froi.i  the  arciiival  update  recordings.  The  SAVE 
DATA  BASE  commands  will  be  used  to  perform  these  backup  procedures. 
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TADB  Record  Structure 


3ARB30630 


table  G-2 

TADB  RECORD  DEFINITION 


1  Course  Number  (key  name  X(12)); 

2  STS/CTS  number  (key  name  X(06))- 

3  reference  group  (rg);  ’ 

10  STS/CTS  reference 

11  duty 

12  date 

13  analyst 

14  proficiency  code 

15  task  group  (rg  in  3  ); 

100  STS  task  stater^ent 

101  betiavior 

102  condition 

103  standards 

104  references 

105  step  group  (rg  in  15); 

200  step  number 

201  subtask/decision 

202  deci sion  yes 

203  decision  no 

204  go  to  step 

205  ski  1 1 s-i nventory 

300  skills 
501  knowledge 


max  200  occurrence 
r.iax  200  occurrence 

(key  name  X(06)  in  3); 

(key  name  X(IOO)  in  3); 

(non-key  date  in  3); 

(non-key  name  X(25)  in  3); 

(key  name  X(02 )  in  3 ); 

(key  name  X(IOO)  in  15); 

(key  name  X(lOo)  in  15); 

(key  name  X(IOO)  in  15); 

(key  nar,ie  X(lljU)  in  15); 

(key  name  X(IUO)  in  15); 

(non-key  integer  9(02)  in  105); 
(key  name  X(75)  in  105); 
(non-key  integer  9(U2)  in  105); 
non-key  integer  9(02)  in  105),’ 
(non-key  integer  9(02)  in  105); 
group  (rg  in  105); 

(key  name  X(50)  in  205  ); 

(key  name  X(50)  in  205  ); 

814  characters/entry 


These  recordings  are  kept  on  the  Update  File  as  directed  by  the 
user.  The  Update  File  is  established  by  specifying  the  Update  File 
tape  identification  when  either  a  LOAD  or  SAVE  Data  Base  coininand  is 
issued.  The  tape  identification  maintains  the  relationship  between 
the  archival  data  base  saved  on  tape  and  the  associated  Update  File. 

With  the  use  of  the  Update  File,  a  user  may  keep  the  archival  data 
base  on  one  tape  and  the  update  commands  on  another  tape,  such  that 
when  the  cycles  recorded  on  the  Update  File  are  applied  to  the  archival 
data  base,  the  resultant  cycle  number  of  the  working  data  base  is 
advanced  from  that  of  the  cycle  on  the  archival  data  base.  The  data 
cycle  number  of  each  copy  will  be  the  same.  The  archival  TADB  and 
Update  File  may  be  used  to  restore  the  data  base  to  current  status  when 
required.  Additionally,  the  data  base  can  be  copied  from  the  random 
access  device  to  tape  daily  as  part  of  tl:e  standard  operating  installa¬ 
tion  backup  procedure. 

Operational  Scenario  -  System  2000  provides  interactive  query 
and  search  capability  geared  for  use  by  non-programmers.  However,  the 
syntax  of  the  system  is  relatively  complex  and  often  obscure  to  ttie 
non-programmer  user.  A  PLI  program  will  be  written  to  drive  the  data 
entry  facilities  of  the  system.  Task  analysts  (non-prograr.imers )  will  be 
prompted  on  an  item  by  item  basis  for  input.  Data  response  will  be 
thoroughly  edited  and  validated  for  corrections  within  the  constraints 
of  each  data  element.  Invalid  data  entries  will  be  reported  to  tlie 
analyst  with  a  message  indicating  the  type  of  error.  The  analyst  will 
be  reprompted  for  the  correct  entry.  The  process  will  continue  until 
a  valid  entry  is  input.  Prompting  will  preserve  the  hierarchical  data 
base  structure  --  course  number  must  be  established  prior  to  entry  of 
dependent  components.  Data  entry  will  be  performed  interactively  in 
System  2000  Immediate  Access  mode.  Entry  sequences  will  be  as  shown  in 
the  following  examples. 
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D/'TA  ENTRY/UPDATE  -  Example  1 


ARE  YOU  ENTERING  A  NEW  COURSE? 

Yes  (or  carriage  return) 

ENTER  COURSE  NUMBER: 

3ABR30630 

ENTER  STS/CTS  REFEREIXE: 

12b(4) 

ENTER  DUTY: 

Maintenance  of  Cryptographic  Equipment 


ENTER  KNOWLEDGE: 
Location  of  fuse 


DATA  ENTRY/UPDATE  -  Example  2 

ARE  YOU  ENTERING  A  NEW  COURSE? 

Yes  (or  carriage  return) 

ENTER  COURSE  NUMBER; 

ABR90330 

**INVALID  COURSE  NUMBER  SYNTAX 
ENTER  COURSE  NUMBER: 

3ABR90330 

ENTER  STS/CTS  NUMBER: 

9030 

**INVALID  STS/CTS  NUMBER  SYNTAX 
ENTER  STS/CTS  NUMBER: 

903X0 

ENTER  STS/CTS  REFERENCE: 

15(6) 


/ 

•  / 

•  / 

ENTER  KNOWLEDGE: 

Correct  positioning  procedure 
ARE  YOU  ENTERING  A  NEW  COURSE? 

Stop  (STOP  may  be  entered  for  any  proi.ipt 

to  discontinue  entry  for  a  course 
— EXIT —  number) 
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DATA  ENTRY/UPDATE  -  Example  ~3 

ARE  YOU  ENTERING  A  NEW  COURSE? 

Yes  (or  carriage  return) 

ENTER  COURSE  NUMBER: 

3ABR30630 

COURSE  NUMBER  EXISTS  IN  DATA  BASE 
ENTER  COURSE  NUMBER: 

3ABR64530 


Data  base  update  requests  v/ill  be  performed  in  the  same  operational 
mode  as  data  entry.  Analysts  will  be  prompted  for  input  through  an 
interactive  PLI  program.  Inputs  will  be  edited  to  maintain  data  base 
integrity.  No  dependent  element  may  be  accessed  without  proper  iden¬ 
tification  of  the  parent  component  (course  number).  Invalid  analyst 
responses  will  be  reported  back  to  the  user  for  re-entry.  Data  update 
will  be  performed  interactively  in  System  2000  Immediate  Access  mode  as 
shown  in  the  following  example. 


ARE  YOU  ENTERING  A  NEW  COURSE? 

No 

ENTER  COURSE  NUMBER: 

3ABR30630 

ENTER  COMPONENT  ITEM  TO  BE  UPDATED: 

Analyst 

ENTER  STS/CTS  REFERENCE: 

12b(4) 

ENTER  ANALYST: 

Stephenson 

ENTER  COMPONENT  ITEM  TO  BE  UPDATED: 

Behavior 

ENTER  STS  TASK  STATEMENT: 

Isolate  defective  detailed  parts  in/on  power  supply 
circuits 
ENTER  BEHAVIOR: 

Same  (this  response  will  duplicate  the 

previous  entry) 

ENTER  COMPONENT  ITEM  TO  BE  UPDATED: 

Subtask 

ENTER  STEP  NUMBER: 

12 

**STEP  NUMBER  12  NOT  FOUND 
ENTER  STEP  NUMBER: 

1 

ENTER  SUBTASK: 

Check  fuse  for  continuity  with  multimeter 
ENTER  COMPONENT  ITEM  TO  BE  UPDATED: 

Stop 


Queries  upon  the  data  base  will  also  be  accompl  i stied  through  an 
interactive  PLI  program.  The  proposed  program  will  enhance  the  current 
query  and  search  capabilities  of  System  2000  and  enable  the  analyst  to 
refine  the  search/selection  process.  Search  requests  will  be  processed 
interactively.  The  number  of  selected  occurrences  ("hits")  will  be 
reported  to  the  analyst,  who  may  then  issue  recursive  requests  effec¬ 
tively  narrowing  (refining)  search  criteria.  Requests  to  print  the 
selected  data  may  be  issued  at  any  point  during  the  search.  Enhanced 
key  word  processing  techniques  will  also  be  developed  to  permit  more 
efficient  and  effective  use  of  complex  keyword  selector  components. 
Inquiry  sequences  will  be  as  shown  in  the  following  examples. 
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DATA  INQUIRY  -  Example"! 


ENTER  COMPONENT  ITEM  TO  BE  SEARCHED: 

Duty 

ENTER  DUTY  TO  BE  LOCATED: 

Maiatenance  of  Cryptographic  Equipinent 
ENTER  COMPONENT  ITEM  TO  BE  SEARCHED: 

Stop 

**6  ITEMS  LOCATED 

DO  YOU  WANT  TO  PRINT  LOCATED  ITEMS? 

No 

ENTER  DUTY  TO  BE  LOCATED: 

Same  (this  response  will  duplicate  the 

previous  entry  for  Duty  selection) 

ENTER  COMPONENT  ITEM  TO  BE  SEARCHED: 

Proficiency  code 

ENTER  PROFICIENCY  CODE  TO  BE  LOCATED: 
la,  lb 

ENTER  COMPONENT  ITEM  TO  BE  SEARCHED: 

Stop 

**2  ITEMS  LOCATED 

DO  YOU  WANT  TO  PRINT  LOCATED  ITEMS?  Option: 

Yes  Selected  items  may 

be  autofnatical ly 
printed  if  less 
than  5 

ENTER  COMPONENT  ITEMS  TO  BE  PRINTED: 

Course  number,  STS/CTS  number.  Duty,  Proficiency  Code 
(enter  ALL  for  complete  list  of  selected  data  structure) 
DO  YOU  WANT  HARD-COPY  (H),  TERMINAL  DISPLAY  (T),  OR 
BOTH  (B)? 

A 

**INVALID  ROUTING  CODE 

DO  YOU  WANT  HARD-COPY  (H),  TERMINAL  DISPLAY  (T),  OR 
BOTH  (B)? 

T 

Tenninal  display  of  selected  data  items  will  occur.  Print 
fonnat  will  follow  response  to  coniponent  itenis  to  be  printed. 

ENTER  COMPONENT  ITEM  TO  BE  SEARCHED: 

Stop 
— EXIT— 
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lUIRY  -  EXAMPLE  2 


ENTER  COMPONENT  ITEM  TO  BE  SEARCHED: 

Skills 

ENTER  SKILLS  TO  BE  LOCATED: 

Use  of  multimeter 

ENTER  COMPONENT  ITEM  TO  BE  SEARCHED: 

Stop 

**  12  ITEMS  LOCATED 

DO  YOU  WANT  TO  PRINT  LOCATED  ITEMS? 

Yes 

ENTER  COMPONENT  ITEMS  TO  BE  PRINTED: 

Course  number,  STS/CTS  number.  Duty,  Proficiency 
Code,  STS  Task  Statement,  Step  Group,  Skills,  Know¬ 
ledge  (enter  ALL  for  complete  list  of  selected  data 
structure ) 

DO  YOU  WANT  HARD-COPY  (H),  TERMINAL  DISPLAY  (T),  OR  BOTH  (B)? 
T 

Terminal  display  of  selected  data  itans  will  occur. 

Print  fonnat  will  follow  response  to  component  items  to 
be  printed. 

ENTER  COMPONENT  ITEM  TO  BE  SEARCHED: 

Stop 

— EXIT — 


Investment  Analyses 

Preliminary  analyses  indicated  that  the  initial  investment  associ¬ 
ated  with  a  full  TADB  network  was  approximately  2.5  times  as  great  as  that 
associated  with  a  limited  TAOB  network.  The  notion  of  a  limited  network 
to  support  concept  demonstration  and  evaluation  Is  appealing  for  three 
reasons.  First,  up-front  costs  can  be  minimized.  Second,  the  utility  of 
the  TAOB  can  be  empirically  evaluated.  Third,  operational  procedures  can 
be  refined  and  implementation  plans  formulated  prior  to  the  commitment  of 
additional  resources.  Requirements  for  a  TAOB  concept  demonstration  are 
minimal.  Three  remote  sites  (Technical  Training  Centers),  each  with  one 
resident  technical  training  course  scheduled  for  substantive  revision, 
would  provide  a  sufficient  TAOB  test  Bed. 

Conclusion  and  Recommendation 


Preliminary  analyses  indicate  that  a  TAOB  can  be  implemented 
using  state-of-the-art  hardware  and  an  operationally  proven  software 
package.  It  is  expected  that  the  potential  cost  reduction  and  avoidance 
benefits  associated  with  implementing  the  TADB  far  outweigh  the  modest 
expenditure  required  to  support  concept  demonstration  and  evaluation. 
It  is  recommended,  therefore,  that  the  AF  consider  implementation  of  a 
limited  TADB  network  as  a  first  step  in  assessing  the  feasibility  and 
utility  of  a  full  TADB  network. 
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LIST  OF  ACROMYMNS 


AF 

Air  Force 

AFB 

Air  Force  Ease 

AFHRL 

Air  Force  Human  Resources  Laboratory 

AFM 

Air  Force  Manual 

AFP 

Air  Force  Pamphlet 

ATC 

Air  Training  Command 

CO  DAP 

Ca.iprehensive  Occupational  Data  Analysis 

Prograias 

CRI 

Criterion-Referenced  Instruction 

CTS 

Course  Training  Standard 

FER 

Field  Evaluation  Reports 

IPISD 

Interservice  Procedures  for  Instructional 
Development 

System 

ISD 

Instructional  System  Design 

ISRS 

Information  Storage  and  Retrieval  System 

MAC 

Military  Aircraft  Command 

MAJCOM 

Major  Command 

MDAC-S 

McDonnell  Douglas  Astronautics  Company  - 

St.  Louis 

OJT 

On-the-Job  Training 

OPiC 

Occupational  Measurement  Center 

OS 

Occupational  Survey 

OSR 

Occupational  Survey  Report 

PCO 

Preliminary  Criterion  Objective 

PLI 

Procedural  Language  Interface 

POI 

Plan  of  Instruction 

PPR 

Preliminary  Peformance  Requirenient 

k  dnd  D 

Research  and  Developnient 

SAC 

Strategic  Air  Command 

SMS 

Subject  Matter  Specialist 

STS 

Speciality  Training  Standard 

TAC 

Tactical  Air  Command 

TADE 

Task  Analysis  Data  Bank 

TK 

Task  Knowledge 

T.O. 

Technical  Order 

TP 

Task  Perfonnance 

TTC 

Technical  Training  Center 

TTG 

Technical  Training  Group 
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